You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oozie.apache.org by Mohamed Battisha <mb...@yahoo.com> on 2011/12/06 22:31:34 UTC

Review Request: OOZIE-591: Oozie continues to materialize new actions after end date modification

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3030/
-----------------------------------------------------------

Review request for oozie.


Summary
-------

Oozie continues to materialize new actions after end date modification. The main issue is related how Oozie materialized the actions based on a FIFO priority queue. Changing the status of a bundle/coordinator job should take higher priority than executing this job. 

The main idea is to enable the queue to handle interruptions. Mainly, once you finished what you are working on currently, you should focus on executing this next action. 

The issue can be illustrated as follow: 

1. Configure a pipeline to run for 1 hour 
2. Start the pipeline 
3. After it starts materializing new actions, change the end time (in my example - to 10 minutes after the pipeline 
starts) 
4. Monitor the coordinator apps - they will continue to materialize new actions past the end time.


This addresses bug OOZIE-591.
    https://issues.apache.org/jira/browse/OOZIE-591


Diffs
-----

  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/Command.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/XCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobResumeXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePauseXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleRerunXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStartXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStatusUpdateXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionStartXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestCallableQueueService.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionInputCheckXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/XCallable.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/resources/oozie-default.xml 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleUnpauseXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionNotificationXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionReadyXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionStartXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionTimeOutXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdateXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPauseXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordRerunXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordResumeXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSuspendXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordUnpauseXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ResumeXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WfEndXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/RecoveryService.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/PollablePriorityDelayQueue.java 1209829 

Diff: https://reviews.apache.org/r/3030/diff


Testing
-------

Regression for all the Commands in addition for specific testing for the Interrupt Driven Map


Thanks,

Mohamed


Re: Review Request: OOZIE-591: Oozie continues to materialize new actions after end date modification

Posted by Mohamed Battisha <mb...@yahoo.com>.

> On 2011-12-08 22:51:31, Mohammad Islam wrote:
> > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java, line 264
> > <https://reviews.apache.org/r/3030/diff/2/?file=62500#file62500line264>
> >
> >     There is a typo in the method name 3 r's. Is it intentional?

fixed


- Mohamed


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3030/#review3757
-----------------------------------------------------------


On 2011-12-09 18:58:56, Mohamed Battisha wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/3030/
> -----------------------------------------------------------
> 
> (Updated 2011-12-09 18:58:56)
> 
> 
> Review request for oozie.
> 
> 
> Summary
> -------
> 
> Oozie continues to materialize new actions after end date modification. The main issue is related how Oozie materialized the actions based on a FIFO priority queue. Changing the status of a bundle/coordinator job should take higher priority than executing this job. 
> 
> The main idea is to enable the queue to handle interruptions. Mainly, once you finished what you are working on currently, you should focus on executing this next action. 
> 
> The issue can be illustrated as follow: 
> 
> 1. Configure a pipeline to run for 1 hour 
> 2. Start the pipeline 
> 3. After it starts materializing new actions, change the end time (in my example - to 10 minutes after the pipeline 
> starts) 
> 4. Monitor the coordinator apps - they will continue to materialize new actions past the end time.
> 
> 
> This addresses bug OOZIE-591.
>     https://issues.apache.org/jira/browse/OOZIE-591
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/Command.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/XCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobResumeXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleRerunXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStartXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStatusUpdateXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleUnpauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionNotificationXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionReadyXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionStartXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionTimeOutXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdateXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordRerunXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordResumeXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSuspendXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordUnpauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ResumeXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WfEndXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/RecoveryService.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/PollablePriorityDelayQueue.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/XCallable.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/resources/oozie-default.xml 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionInputCheckXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionStartXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestCallableQueueService.java 1209829 
> 
> Diff: https://reviews.apache.org/r/3030/diff
> 
> 
> Testing
> -------
> 
> Regression for all the Commands in addition for specific testing for the Interrupt Driven Map
> 
> 
> Thanks,
> 
> Mohamed
> 
>


Re: Review Request: OOZIE-591: Oozie continues to materialize new actions after end date modification

Posted by Mohammad Islam <mi...@yahoo.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3030/#review3757
-----------------------------------------------------------



http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java
<https://reviews.apache.org/r/3030/#comment8490>

    There is a typo in the method name 3 r's. Is it intentional?


- Mohammad


On 2011-12-06 21:31:34, Mohamed Battisha wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/3030/
> -----------------------------------------------------------
> 
> (Updated 2011-12-06 21:31:34)
> 
> 
> Review request for oozie.
> 
> 
> Summary
> -------
> 
> Oozie continues to materialize new actions after end date modification. The main issue is related how Oozie materialized the actions based on a FIFO priority queue. Changing the status of a bundle/coordinator job should take higher priority than executing this job. 
> 
> The main idea is to enable the queue to handle interruptions. Mainly, once you finished what you are working on currently, you should focus on executing this next action. 
> 
> The issue can be illustrated as follow: 
> 
> 1. Configure a pipeline to run for 1 hour 
> 2. Start the pipeline 
> 3. After it starts materializing new actions, change the end time (in my example - to 10 minutes after the pipeline 
> starts) 
> 4. Monitor the coordinator apps - they will continue to materialize new actions past the end time.
> 
> 
> This addresses bug OOZIE-591.
>     https://issues.apache.org/jira/browse/OOZIE-591
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/Command.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/XCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobResumeXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleRerunXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStartXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStatusUpdateXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionStartXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestCallableQueueService.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionInputCheckXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/XCallable.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/resources/oozie-default.xml 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleUnpauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionNotificationXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionReadyXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionStartXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionTimeOutXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdateXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordRerunXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordResumeXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSuspendXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordUnpauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ResumeXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WfEndXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/RecoveryService.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/PollablePriorityDelayQueue.java 1209829 
> 
> Diff: https://reviews.apache.org/r/3030/diff
> 
> 
> Testing
> -------
> 
> Regression for all the Commands in addition for specific testing for the Interrupt Driven Map
> 
> 
> Thanks,
> 
> Mohamed
> 
>


Re: Review Request: OOZIE-591: Oozie continues to materialize new actions after end date modification

Posted by Mohamed Battisha <mb...@yahoo.com>.

> On 2011-12-08 22:46:00, Mohammad Islam wrote:
> > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java, line 414
> > <https://reviews.apache.org/r/3030/diff/2/?file=62500#file62500line414>
> >
> >     extra space

done


> On 2011-12-08 22:46:00, Mohammad Islam wrote:
> > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java, line 93
> > <https://reviews.apache.org/r/3030/diff/2/?file=62500#file62500line93>
> >
> >     ConcurrentHashMap might be better data structure.

done


> On 2011-12-08 22:46:00, Mohammad Islam wrote:
> > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java, line 507
> > <https://reviews.apache.org/r/3030/diff/2/?file=62500#file62500line507>
> >
> >     minor thing : type exit-->exist

done


> On 2011-12-08 22:46:00, Mohammad Islam wrote:
> > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java, line 728
> > <https://reviews.apache.org/r/3030/diff/2/?file=62500#file62500line728>
> >
> >     the whole list is being used. so the the list for specific lockKey should be removed. I think code is right. the comments needs to be updated.

done


> On 2011-12-08 22:46:00, Mohammad Islam wrote:
> > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java, line 764
> > <https://reviews.apache.org/r/3030/diff/2/?file=62500#file62500line764>
> >
> >     the log level should be "warn" instead of "trace"

done


> On 2011-12-08 22:46:00, Mohammad Islam wrote:
> > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/PollablePriorityDelayQueue.java, line 53
> > <https://reviews.apache.org/r/3030/diff/2/?file=62502#file62502line53>
> >
> >     The code between line 53-59 and 72-79 looks very similar. Can we make it to a separate method and call it from this two places?

I thought to do as a single method but there are two many moving part that are updated in these few lines, which will not make it as clean


> On 2011-12-08 22:46:00, Mohammad Islam wrote:
> > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/resources/oozie-default.xml, line 286
> > <https://reviews.apache.org/r/3030/diff/2/?file=62504#file62504line286>
> >
> >     What about suspend and pause command?

It just used samples of the commands. I believe it should be changed based on the deployment. that siad, I will add pause and suspend as well


- Mohamed


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3030/#review3756
-----------------------------------------------------------


On 2011-12-06 21:31:34, Mohamed Battisha wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/3030/
> -----------------------------------------------------------
> 
> (Updated 2011-12-06 21:31:34)
> 
> 
> Review request for oozie.
> 
> 
> Summary
> -------
> 
> Oozie continues to materialize new actions after end date modification. The main issue is related how Oozie materialized the actions based on a FIFO priority queue. Changing the status of a bundle/coordinator job should take higher priority than executing this job. 
> 
> The main idea is to enable the queue to handle interruptions. Mainly, once you finished what you are working on currently, you should focus on executing this next action. 
> 
> The issue can be illustrated as follow: 
> 
> 1. Configure a pipeline to run for 1 hour 
> 2. Start the pipeline 
> 3. After it starts materializing new actions, change the end time (in my example - to 10 minutes after the pipeline 
> starts) 
> 4. Monitor the coordinator apps - they will continue to materialize new actions past the end time.
> 
> 
> This addresses bug OOZIE-591.
>     https://issues.apache.org/jira/browse/OOZIE-591
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/Command.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/XCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobResumeXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleRerunXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStartXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStatusUpdateXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionStartXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestCallableQueueService.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionInputCheckXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/XCallable.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/resources/oozie-default.xml 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleUnpauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionNotificationXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionReadyXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionStartXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionTimeOutXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdateXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordRerunXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordResumeXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSuspendXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordUnpauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ResumeXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WfEndXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/RecoveryService.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/PollablePriorityDelayQueue.java 1209829 
> 
> Diff: https://reviews.apache.org/r/3030/diff
> 
> 
> Testing
> -------
> 
> Regression for all the Commands in addition for specific testing for the Interrupt Driven Map
> 
> 
> Thanks,
> 
> Mohamed
> 
>


Re: Review Request: OOZIE-591: Oozie continues to materialize new actions after end date modification

Posted by Mohammad Islam <mi...@yahoo.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3030/#review3756
-----------------------------------------------------------


Overall clean implementations.
We need to give higher priority to all the high priority command.
Consider a case: where there is only one ChangeCommand for a job. but there are other commands from other jobs. If change command doesn't have the higher priority, it have wait for its turn. Highest priority will give it a better chance.

Kill command is already high priority (2) command (See CoordKillXCommand). We need to do the similar things for change/suspend/pause commands. Also make sure the same is true for WF/Coord/Bundle*XCommand level.


http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java
<https://reviews.apache.org/r/3030/#comment8484>

    ConcurrentHashMap might be better data structure.



http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java
<https://reviews.apache.org/r/3030/#comment8485>

    extra space



http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java
<https://reviews.apache.org/r/3030/#comment8486>

    minor thing : type exit-->exist



http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java
<https://reviews.apache.org/r/3030/#comment8487>

    the whole list is being used. so the the list for specific lockKey should be removed. I think code is right. the comments needs to be updated.



http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java
<https://reviews.apache.org/r/3030/#comment8488>

    the log level should be "warn" instead of "trace"



http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/PollablePriorityDelayQueue.java
<https://reviews.apache.org/r/3030/#comment8489>

    The code between line 53-59 and 72-79 looks very similar. Can we make it to a separate method and call it from this two places? 



http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/resources/oozie-default.xml
<https://reviews.apache.org/r/3030/#comment8483>

    What about suspend and pause command?


- Mohammad


On 2011-12-06 21:31:34, Mohamed Battisha wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/3030/
> -----------------------------------------------------------
> 
> (Updated 2011-12-06 21:31:34)
> 
> 
> Review request for oozie.
> 
> 
> Summary
> -------
> 
> Oozie continues to materialize new actions after end date modification. The main issue is related how Oozie materialized the actions based on a FIFO priority queue. Changing the status of a bundle/coordinator job should take higher priority than executing this job. 
> 
> The main idea is to enable the queue to handle interruptions. Mainly, once you finished what you are working on currently, you should focus on executing this next action. 
> 
> The issue can be illustrated as follow: 
> 
> 1. Configure a pipeline to run for 1 hour 
> 2. Start the pipeline 
> 3. After it starts materializing new actions, change the end time (in my example - to 10 minutes after the pipeline 
> starts) 
> 4. Monitor the coordinator apps - they will continue to materialize new actions past the end time.
> 
> 
> This addresses bug OOZIE-591.
>     https://issues.apache.org/jira/browse/OOZIE-591
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/Command.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/XCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobResumeXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleRerunXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStartXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStatusUpdateXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionStartXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestCallableQueueService.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionInputCheckXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/XCallable.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/resources/oozie-default.xml 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleUnpauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionNotificationXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionReadyXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionStartXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionTimeOutXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdateXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordRerunXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordResumeXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSuspendXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordUnpauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ResumeXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WfEndXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/RecoveryService.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/PollablePriorityDelayQueue.java 1209829 
> 
> Diff: https://reviews.apache.org/r/3030/diff
> 
> 
> Testing
> -------
> 
> Regression for all the Commands in addition for specific testing for the Interrupt Driven Map
> 
> 
> Thanks,
> 
> Mohamed
> 
>


Re: Review Request: OOZIE-591: Oozie continues to materialize new actions after end date modification

Posted by Alejandro Abdelnur <tu...@cloudera.com>.

> On 2011-12-12 20:44:14, Alejandro Abdelnur wrote:
> > The proposed approach seems overkilling when the following change would achieve the same without introducing new concurrent logic:
> > 
> > * The CallableQueueService would define a PriorityQueue with 4 priorities instead of 3.
> > * The queue() method would ensure there no command being queued has priority 4 (forbidding direct use of the the highest priority).
> > * The CallableQueueService would a new protected method 'Set<Class> getInterruptCommandClasses()' that would return the commands that are 'interrupt' commands.
> > * On Queuing, within the queue() command any command which class is in the above Set will be bumped to priority 4.
> > 
> > This achieves exactly the same behavior with the current well tested logic.
> >
> 
> Mohamed Battisha wrote:
>     I do not see it overkilling. It is exactly the same behavior you proposed but instead of using a high priority queue, we are using a per-job interrupt map. I tried as much as possible to implement it within the same theme. [I am mapping the poposed behavior with the current design which is almost 1:1]
>     - * The CallableQueueService would define a PriorityQueue with 4 priorities instead of 3.
>     - Using a InterruptMap within the CallableQueueService.
>     
>     * The queue() method would ensure there no command being queued has priority 4 (forbidding direct use of the the highest priority).
>     - No change here are we are not going to use a special queue. instead while polling the data from the queue, we will check the map first
>     
>     * The CallableQueueService would a new protected method 'Set<Class> getInterruptCommandClasses()' that would return the commands that are 'interrupt' commands.
>     * The interrupt commands are stored in a local set within CallableQueueService
>     
>     * On Queuing, within the queue() command any command which class is in the above Set will be bumped to priority 4.
>     On Queuing, within the queue command, if the command type belongs to the InterruptSet, it will inserted to Interrupt Map
>     
>     
>     The major advantage of the map approach over queue approach is we are a avoiding job starvation. Specially, in large deployment in which hundreds of thousands of commands are running from thousands of jobs. In this case a few jobs with interrupt commands may take full control of the thread poll causing other jobs to starve. The per-job map will avoid such a behavior
> 
> Alejandro Abdelnur wrote:
>     Mohamed, 
>     
>     The QueueCommandService has anti-starvation logic already, handled by the callable.concurrency settings. This ensures that never a single command takes over all threads. Because of this mechanism, what you point out as an issue, it is not.
>     
>     So again, the alternate solution I'm suggesting restricts the changes to one place (the CallableQueueService) and it uses the existing, well proven priority and concurrency mechanism handled by the CllableQueueService.
>     
>     Thanks.
>     
>     Alejandro
>
> 
> Mohamed Battisha wrote:
>     Alejandro
>     I am assumed from your solution, the anti-starvation will not push other commands to the higher priority queue which will be only restricted for the interrupted commands
>     If the anti-starvation will push other commands to the higher priority queue, so we are facing the same exact issue again. we are only using four queues instead of three.
>     
>     The changes I made in mainly at the CallableQueueService with the same exact logic you proposed [using a map instead of a higher priority queue]. During my tests, i figured out there are some commands that are not checking for null ref in GetLockKey(). I added these checks in these commands to avoid a null-ref exceptions in some rare condition which give an impression of a bigger change :)
>     
>     Thanks!
>     Mohamed

Mohamed,

Your are correct, then antiStarvation() method in the PriorityDelayQueue should be modified to stop the priority rising one priority less this means that line 488 should do:

            for (int i = 0; i < queues.length - 2; i++) {

Good catch.

Thxs.


- Alejandro


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3030/#review3851
-----------------------------------------------------------


On 2011-12-09 18:58:56, Mohamed Battisha wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/3030/
> -----------------------------------------------------------
> 
> (Updated 2011-12-09 18:58:56)
> 
> 
> Review request for oozie.
> 
> 
> Summary
> -------
> 
> Oozie continues to materialize new actions after end date modification. The main issue is related how Oozie materialized the actions based on a FIFO priority queue. Changing the status of a bundle/coordinator job should take higher priority than executing this job. 
> 
> The main idea is to enable the queue to handle interruptions. Mainly, once you finished what you are working on currently, you should focus on executing this next action. 
> 
> The issue can be illustrated as follow: 
> 
> 1. Configure a pipeline to run for 1 hour 
> 2. Start the pipeline 
> 3. After it starts materializing new actions, change the end time (in my example - to 10 minutes after the pipeline 
> starts) 
> 4. Monitor the coordinator apps - they will continue to materialize new actions past the end time.
> 
> 
> This addresses bug OOZIE-591.
>     https://issues.apache.org/jira/browse/OOZIE-591
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/Command.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/XCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobResumeXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleRerunXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStartXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStatusUpdateXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleUnpauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionNotificationXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionReadyXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionStartXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionTimeOutXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdateXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordRerunXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordResumeXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSuspendXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordUnpauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ResumeXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WfEndXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/RecoveryService.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/PollablePriorityDelayQueue.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/XCallable.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/resources/oozie-default.xml 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionInputCheckXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionStartXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestCallableQueueService.java 1209829 
> 
> Diff: https://reviews.apache.org/r/3030/diff
> 
> 
> Testing
> -------
> 
> Regression for all the Commands in addition for specific testing for the Interrupt Driven Map
> 
> 
> Thanks,
> 
> Mohamed
> 
>


Re: Review Request: OOZIE-591: Oozie continues to materialize new actions after end date modification

Posted by Mohamed Battisha <mb...@yahoo.com>.

> On 2011-12-12 20:44:14, Alejandro Abdelnur wrote:
> > The proposed approach seems overkilling when the following change would achieve the same without introducing new concurrent logic:
> > 
> > * The CallableQueueService would define a PriorityQueue with 4 priorities instead of 3.
> > * The queue() method would ensure there no command being queued has priority 4 (forbidding direct use of the the highest priority).
> > * The CallableQueueService would a new protected method 'Set<Class> getInterruptCommandClasses()' that would return the commands that are 'interrupt' commands.
> > * On Queuing, within the queue() command any command which class is in the above Set will be bumped to priority 4.
> > 
> > This achieves exactly the same behavior with the current well tested logic.
> >

I do not see it overkilling. It is exactly the same behavior you proposed but instead of using a high priority queue, we are using a per-job interrupt map. I tried as much as possible to implement it within the same theme. [I am mapping the poposed behavior with the current design which is almost 1:1]
- * The CallableQueueService would define a PriorityQueue with 4 priorities instead of 3.
- Using a InterruptMap within the CallableQueueService.

* The queue() method would ensure there no command being queued has priority 4 (forbidding direct use of the the highest priority).
- No change here are we are not going to use a special queue. instead while polling the data from the queue, we will check the map first

* The CallableQueueService would a new protected method 'Set<Class> getInterruptCommandClasses()' that would return the commands that are 'interrupt' commands.
* The interrupt commands are stored in a local set within CallableQueueService

* On Queuing, within the queue() command any command which class is in the above Set will be bumped to priority 4.
On Queuing, within the queue command, if the command type belongs to the InterruptSet, it will inserted to Interrupt Map


The major advantage of the map approach over queue approach is we are a avoiding job starvation. Specially, in large deployment in which hundreds of thousands of commands are running from thousands of jobs. In this case a few jobs with interrupt commands may take full control of the thread poll causing other jobs to starve. The per-job map will avoid such a behavior


- Mohamed


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3030/#review3851
-----------------------------------------------------------


On 2011-12-09 18:58:56, Mohamed Battisha wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/3030/
> -----------------------------------------------------------
> 
> (Updated 2011-12-09 18:58:56)
> 
> 
> Review request for oozie.
> 
> 
> Summary
> -------
> 
> Oozie continues to materialize new actions after end date modification. The main issue is related how Oozie materialized the actions based on a FIFO priority queue. Changing the status of a bundle/coordinator job should take higher priority than executing this job. 
> 
> The main idea is to enable the queue to handle interruptions. Mainly, once you finished what you are working on currently, you should focus on executing this next action. 
> 
> The issue can be illustrated as follow: 
> 
> 1. Configure a pipeline to run for 1 hour 
> 2. Start the pipeline 
> 3. After it starts materializing new actions, change the end time (in my example - to 10 minutes after the pipeline 
> starts) 
> 4. Monitor the coordinator apps - they will continue to materialize new actions past the end time.
> 
> 
> This addresses bug OOZIE-591.
>     https://issues.apache.org/jira/browse/OOZIE-591
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/Command.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/XCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobResumeXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleRerunXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStartXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStatusUpdateXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleUnpauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionNotificationXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionReadyXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionStartXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionTimeOutXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdateXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordRerunXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordResumeXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSuspendXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordUnpauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ResumeXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WfEndXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/RecoveryService.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/PollablePriorityDelayQueue.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/XCallable.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/resources/oozie-default.xml 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionInputCheckXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionStartXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestCallableQueueService.java 1209829 
> 
> Diff: https://reviews.apache.org/r/3030/diff
> 
> 
> Testing
> -------
> 
> Regression for all the Commands in addition for specific testing for the Interrupt Driven Map
> 
> 
> Thanks,
> 
> Mohamed
> 
>


Re: Review Request: OOZIE-591: Oozie continues to materialize new actions after end date modification

Posted by Mohamed Battisha <mb...@yahoo.com>.

> On 2011-12-12 20:44:14, Alejandro Abdelnur wrote:
> > The proposed approach seems overkilling when the following change would achieve the same without introducing new concurrent logic:
> > 
> > * The CallableQueueService would define a PriorityQueue with 4 priorities instead of 3.
> > * The queue() method would ensure there no command being queued has priority 4 (forbidding direct use of the the highest priority).
> > * The CallableQueueService would a new protected method 'Set<Class> getInterruptCommandClasses()' that would return the commands that are 'interrupt' commands.
> > * On Queuing, within the queue() command any command which class is in the above Set will be bumped to priority 4.
> > 
> > This achieves exactly the same behavior with the current well tested logic.
> >
> 
> Mohamed Battisha wrote:
>     I do not see it overkilling. It is exactly the same behavior you proposed but instead of using a high priority queue, we are using a per-job interrupt map. I tried as much as possible to implement it within the same theme. [I am mapping the poposed behavior with the current design which is almost 1:1]
>     - * The CallableQueueService would define a PriorityQueue with 4 priorities instead of 3.
>     - Using a InterruptMap within the CallableQueueService.
>     
>     * The queue() method would ensure there no command being queued has priority 4 (forbidding direct use of the the highest priority).
>     - No change here are we are not going to use a special queue. instead while polling the data from the queue, we will check the map first
>     
>     * The CallableQueueService would a new protected method 'Set<Class> getInterruptCommandClasses()' that would return the commands that are 'interrupt' commands.
>     * The interrupt commands are stored in a local set within CallableQueueService
>     
>     * On Queuing, within the queue() command any command which class is in the above Set will be bumped to priority 4.
>     On Queuing, within the queue command, if the command type belongs to the InterruptSet, it will inserted to Interrupt Map
>     
>     
>     The major advantage of the map approach over queue approach is we are a avoiding job starvation. Specially, in large deployment in which hundreds of thousands of commands are running from thousands of jobs. In this case a few jobs with interrupt commands may take full control of the thread poll causing other jobs to starve. The per-job map will avoid such a behavior
> 
> Alejandro Abdelnur wrote:
>     Mohamed, 
>     
>     The QueueCommandService has anti-starvation logic already, handled by the callable.concurrency settings. This ensures that never a single command takes over all threads. Because of this mechanism, what you point out as an issue, it is not.
>     
>     So again, the alternate solution I'm suggesting restricts the changes to one place (the CallableQueueService) and it uses the existing, well proven priority and concurrency mechanism handled by the CllableQueueService.
>     
>     Thanks.
>     
>     Alejandro
>
> 
> Mohamed Battisha wrote:
>     Alejandro
>     I am assumed from your solution, the anti-starvation will not push other commands to the higher priority queue which will be only restricted for the interrupted commands
>     If the anti-starvation will push other commands to the higher priority queue, so we are facing the same exact issue again. we are only using four queues instead of three.
>     
>     The changes I made in mainly at the CallableQueueService with the same exact logic you proposed [using a map instead of a higher priority queue]. During my tests, i figured out there are some commands that are not checking for null ref in GetLockKey(). I added these checks in these commands to avoid a null-ref exceptions in some rare condition which give an impression of a bigger change :)
>     
>     Thanks!
>     Mohamed
> 
> Alejandro Abdelnur wrote:
>     Mohamed,
>     
>     Your are correct, then antiStarvation() method in the PriorityDelayQueue should be modified to stop the priority rising one priority less this means that line 488 should do:
>     
>                 for (int i = 0; i < queues.length - 2; i++) {
>     
>     Good catch.
>     
>     Thxs.

I did some experiment to compare the two approached [Interrupt Map  Vs. the Hi-Pri Queue]

The goal of the experiment is to figure out how the Hi-Pri Queue reacted to a long locking time for a few jobs.
In the experiment, I had 10 running jobs on a Thread poll of 5 threads. Out of the 10 jobs, 3 of them have a long running commands that required the job to be locked for 5 minutes. 
At the same time, I generated three different interrupt commands for the given three jobs. 

Here what will happen based on the  Hi-Pri Queue approach:
- The interrupt commands will be solely  reside in the Hi-Pri Queue with no other commands from other jobs [no command will be promoted to the Hi-Pri queue].
- 3 of the 5 threads will be mainly allocated to the long running commands.
- The remaining 2 threads will be running on Hi-Pri queue. Mainly trying to acquire the lock, failed and re-instered again on the Hi-Pri Queue [simply doing no thing]

Based on this, the system will be mainly locked for 5 minutes, and the remaining 7 jobs will be simply stalled. 

As I mentioned before, the problem will be exaggerated when the number of jobs increased and its probability and so its frequency will increase.
As well, if some of the commands faces a locking issues which erroneously did not release the lock, it can cause the whole system to be stalled which should be avoided [in shared queue we can not let an erroneous jobs impact the whole queue. our design should isolate is as much as possible]

Using, the interrupt map will simply avoid such a locking scheme, isolating the erroneous jobs while achieving the same goal of giving some commands an immediate execution. 

Again, as I mentioned before,  the change for the interrupt map is pretty simple and we should not be afraid of it. As well, I run a full test cases to cover different scenario and things are pretty solid.

Thanks!
Mohamed


- Mohamed


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3030/#review3851
-----------------------------------------------------------


On 2011-12-09 18:58:56, Mohamed Battisha wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/3030/
> -----------------------------------------------------------
> 
> (Updated 2011-12-09 18:58:56)
> 
> 
> Review request for oozie.
> 
> 
> Summary
> -------
> 
> Oozie continues to materialize new actions after end date modification. The main issue is related how Oozie materialized the actions based on a FIFO priority queue. Changing the status of a bundle/coordinator job should take higher priority than executing this job. 
> 
> The main idea is to enable the queue to handle interruptions. Mainly, once you finished what you are working on currently, you should focus on executing this next action. 
> 
> The issue can be illustrated as follow: 
> 
> 1. Configure a pipeline to run for 1 hour 
> 2. Start the pipeline 
> 3. After it starts materializing new actions, change the end time (in my example - to 10 minutes after the pipeline 
> starts) 
> 4. Monitor the coordinator apps - they will continue to materialize new actions past the end time.
> 
> 
> This addresses bug OOZIE-591.
>     https://issues.apache.org/jira/browse/OOZIE-591
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/Command.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/XCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobResumeXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleRerunXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStartXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStatusUpdateXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleUnpauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionNotificationXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionReadyXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionStartXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionTimeOutXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdateXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordRerunXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordResumeXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSuspendXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordUnpauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ResumeXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WfEndXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/RecoveryService.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/PollablePriorityDelayQueue.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/XCallable.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/resources/oozie-default.xml 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionInputCheckXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionStartXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestCallableQueueService.java 1209829 
> 
> Diff: https://reviews.apache.org/r/3030/diff
> 
> 
> Testing
> -------
> 
> Regression for all the Commands in addition for specific testing for the Interrupt Driven Map
> 
> 
> Thanks,
> 
> Mohamed
> 
>


Re: Review Request: OOZIE-591: Oozie continues to materialize new actions after end date modification

Posted by Alejandro Abdelnur <tu...@cloudera.com>.

> On 2011-12-12 20:44:14, Alejandro Abdelnur wrote:
> > The proposed approach seems overkilling when the following change would achieve the same without introducing new concurrent logic:
> > 
> > * The CallableQueueService would define a PriorityQueue with 4 priorities instead of 3.
> > * The queue() method would ensure there no command being queued has priority 4 (forbidding direct use of the the highest priority).
> > * The CallableQueueService would a new protected method 'Set<Class> getInterruptCommandClasses()' that would return the commands that are 'interrupt' commands.
> > * On Queuing, within the queue() command any command which class is in the above Set will be bumped to priority 4.
> > 
> > This achieves exactly the same behavior with the current well tested logic.
> >
> 
> Mohamed Battisha wrote:
>     I do not see it overkilling. It is exactly the same behavior you proposed but instead of using a high priority queue, we are using a per-job interrupt map. I tried as much as possible to implement it within the same theme. [I am mapping the poposed behavior with the current design which is almost 1:1]
>     - * The CallableQueueService would define a PriorityQueue with 4 priorities instead of 3.
>     - Using a InterruptMap within the CallableQueueService.
>     
>     * The queue() method would ensure there no command being queued has priority 4 (forbidding direct use of the the highest priority).
>     - No change here are we are not going to use a special queue. instead while polling the data from the queue, we will check the map first
>     
>     * The CallableQueueService would a new protected method 'Set<Class> getInterruptCommandClasses()' that would return the commands that are 'interrupt' commands.
>     * The interrupt commands are stored in a local set within CallableQueueService
>     
>     * On Queuing, within the queue() command any command which class is in the above Set will be bumped to priority 4.
>     On Queuing, within the queue command, if the command type belongs to the InterruptSet, it will inserted to Interrupt Map
>     
>     
>     The major advantage of the map approach over queue approach is we are a avoiding job starvation. Specially, in large deployment in which hundreds of thousands of commands are running from thousands of jobs. In this case a few jobs with interrupt commands may take full control of the thread poll causing other jobs to starve. The per-job map will avoid such a behavior

Mohamed, 

The QueueCommandService has anti-starvation logic already, handled by the callable.concurrency settings. This ensures that never a single command takes over all threads. Because of this mechanism, what you point out as an issue, it is not.

So again, the alternate solution I'm suggesting restricts the changes to one place (the CallableQueueService) and it uses the existing, well proven priority and concurrency mechanism handled by the CllableQueueService.

Thanks.

Alejandro


- Alejandro


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3030/#review3851
-----------------------------------------------------------


On 2011-12-09 18:58:56, Mohamed Battisha wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/3030/
> -----------------------------------------------------------
> 
> (Updated 2011-12-09 18:58:56)
> 
> 
> Review request for oozie.
> 
> 
> Summary
> -------
> 
> Oozie continues to materialize new actions after end date modification. The main issue is related how Oozie materialized the actions based on a FIFO priority queue. Changing the status of a bundle/coordinator job should take higher priority than executing this job. 
> 
> The main idea is to enable the queue to handle interruptions. Mainly, once you finished what you are working on currently, you should focus on executing this next action. 
> 
> The issue can be illustrated as follow: 
> 
> 1. Configure a pipeline to run for 1 hour 
> 2. Start the pipeline 
> 3. After it starts materializing new actions, change the end time (in my example - to 10 minutes after the pipeline 
> starts) 
> 4. Monitor the coordinator apps - they will continue to materialize new actions past the end time.
> 
> 
> This addresses bug OOZIE-591.
>     https://issues.apache.org/jira/browse/OOZIE-591
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/Command.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/XCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobResumeXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleRerunXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStartXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStatusUpdateXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleUnpauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionNotificationXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionReadyXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionStartXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionTimeOutXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdateXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordRerunXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordResumeXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSuspendXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordUnpauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ResumeXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WfEndXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/RecoveryService.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/PollablePriorityDelayQueue.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/XCallable.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/resources/oozie-default.xml 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionInputCheckXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionStartXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestCallableQueueService.java 1209829 
> 
> Diff: https://reviews.apache.org/r/3030/diff
> 
> 
> Testing
> -------
> 
> Regression for all the Commands in addition for specific testing for the Interrupt Driven Map
> 
> 
> Thanks,
> 
> Mohamed
> 
>


Re: Review Request: OOZIE-591: Oozie continues to materialize new actions after end date modification

Posted by Mohamed Battisha <mb...@yahoo.com>.

> On 2011-12-12 20:44:14, Alejandro Abdelnur wrote:
> > The proposed approach seems overkilling when the following change would achieve the same without introducing new concurrent logic:
> > 
> > * The CallableQueueService would define a PriorityQueue with 4 priorities instead of 3.
> > * The queue() method would ensure there no command being queued has priority 4 (forbidding direct use of the the highest priority).
> > * The CallableQueueService would a new protected method 'Set<Class> getInterruptCommandClasses()' that would return the commands that are 'interrupt' commands.
> > * On Queuing, within the queue() command any command which class is in the above Set will be bumped to priority 4.
> > 
> > This achieves exactly the same behavior with the current well tested logic.
> >
> 
> Mohamed Battisha wrote:
>     I do not see it overkilling. It is exactly the same behavior you proposed but instead of using a high priority queue, we are using a per-job interrupt map. I tried as much as possible to implement it within the same theme. [I am mapping the poposed behavior with the current design which is almost 1:1]
>     - * The CallableQueueService would define a PriorityQueue with 4 priorities instead of 3.
>     - Using a InterruptMap within the CallableQueueService.
>     
>     * The queue() method would ensure there no command being queued has priority 4 (forbidding direct use of the the highest priority).
>     - No change here are we are not going to use a special queue. instead while polling the data from the queue, we will check the map first
>     
>     * The CallableQueueService would a new protected method 'Set<Class> getInterruptCommandClasses()' that would return the commands that are 'interrupt' commands.
>     * The interrupt commands are stored in a local set within CallableQueueService
>     
>     * On Queuing, within the queue() command any command which class is in the above Set will be bumped to priority 4.
>     On Queuing, within the queue command, if the command type belongs to the InterruptSet, it will inserted to Interrupt Map
>     
>     
>     The major advantage of the map approach over queue approach is we are a avoiding job starvation. Specially, in large deployment in which hundreds of thousands of commands are running from thousands of jobs. In this case a few jobs with interrupt commands may take full control of the thread poll causing other jobs to starve. The per-job map will avoid such a behavior
> 
> Alejandro Abdelnur wrote:
>     Mohamed, 
>     
>     The QueueCommandService has anti-starvation logic already, handled by the callable.concurrency settings. This ensures that never a single command takes over all threads. Because of this mechanism, what you point out as an issue, it is not.
>     
>     So again, the alternate solution I'm suggesting restricts the changes to one place (the CallableQueueService) and it uses the existing, well proven priority and concurrency mechanism handled by the CllableQueueService.
>     
>     Thanks.
>     
>     Alejandro
>

Alejandro
I am assumed from your solution, the anti-starvation will not push other commands to the higher priority queue which will be only restricted for the interrupted commands
If the anti-starvation will push other commands to the higher priority queue, so we are facing the same exact issue again. we are only using four queues instead of three.

The changes I made in mainly at the CallableQueueService with the same exact logic you proposed [using a map instead of a higher priority queue]. During my tests, i figured out there are some commands that are not checking for null ref in GetLockKey(). I added these checks in these commands to avoid a null-ref exceptions in some rare condition which give an impression of a bigger change :)

Thanks!
Mohamed


- Mohamed


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3030/#review3851
-----------------------------------------------------------


On 2011-12-09 18:58:56, Mohamed Battisha wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/3030/
> -----------------------------------------------------------
> 
> (Updated 2011-12-09 18:58:56)
> 
> 
> Review request for oozie.
> 
> 
> Summary
> -------
> 
> Oozie continues to materialize new actions after end date modification. The main issue is related how Oozie materialized the actions based on a FIFO priority queue. Changing the status of a bundle/coordinator job should take higher priority than executing this job. 
> 
> The main idea is to enable the queue to handle interruptions. Mainly, once you finished what you are working on currently, you should focus on executing this next action. 
> 
> The issue can be illustrated as follow: 
> 
> 1. Configure a pipeline to run for 1 hour 
> 2. Start the pipeline 
> 3. After it starts materializing new actions, change the end time (in my example - to 10 minutes after the pipeline 
> starts) 
> 4. Monitor the coordinator apps - they will continue to materialize new actions past the end time.
> 
> 
> This addresses bug OOZIE-591.
>     https://issues.apache.org/jira/browse/OOZIE-591
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/Command.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/XCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobResumeXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleRerunXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStartXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStatusUpdateXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleUnpauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionNotificationXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionReadyXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionStartXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionTimeOutXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdateXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordRerunXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordResumeXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSuspendXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordUnpauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ResumeXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WfEndXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/RecoveryService.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/PollablePriorityDelayQueue.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/XCallable.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/resources/oozie-default.xml 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionInputCheckXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionStartXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestCallableQueueService.java 1209829 
> 
> Diff: https://reviews.apache.org/r/3030/diff
> 
> 
> Testing
> -------
> 
> Regression for all the Commands in addition for specific testing for the Interrupt Driven Map
> 
> 
> Thanks,
> 
> Mohamed
> 
>


Re: Review Request: OOZIE-591: Oozie continues to materialize new actions after end date modification

Posted by Alejandro Abdelnur <tu...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3030/#review3851
-----------------------------------------------------------


The proposed approach seems overkilling when the following change would achieve the same without introducing new concurrent logic:

* The CallableQueueService would define a PriorityQueue with 4 priorities instead of 3.
* The queue() method would ensure there no command being queued has priority 4 (forbidding direct use of the the highest priority).
* The CallableQueueService would a new protected method 'Set<Class> getInterruptCommandClasses()' that would return the commands that are 'interrupt' commands.
* On Queuing, within the queue() command any command which class is in the above Set will be bumped to priority 4.

This achieves exactly the same behavior with the current well tested logic.


- Alejandro


On 2011-12-09 18:58:56, Mohamed Battisha wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/3030/
> -----------------------------------------------------------
> 
> (Updated 2011-12-09 18:58:56)
> 
> 
> Review request for oozie.
> 
> 
> Summary
> -------
> 
> Oozie continues to materialize new actions after end date modification. The main issue is related how Oozie materialized the actions based on a FIFO priority queue. Changing the status of a bundle/coordinator job should take higher priority than executing this job. 
> 
> The main idea is to enable the queue to handle interruptions. Mainly, once you finished what you are working on currently, you should focus on executing this next action. 
> 
> The issue can be illustrated as follow: 
> 
> 1. Configure a pipeline to run for 1 hour 
> 2. Start the pipeline 
> 3. After it starts materializing new actions, change the end time (in my example - to 10 minutes after the pipeline 
> starts) 
> 4. Monitor the coordinator apps - they will continue to materialize new actions past the end time.
> 
> 
> This addresses bug OOZIE-591.
>     https://issues.apache.org/jira/browse/OOZIE-591
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/Command.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/XCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobResumeXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleRerunXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStartXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStatusUpdateXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleUnpauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionNotificationXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionReadyXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionStartXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionTimeOutXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdateXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordRerunXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordResumeXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSuspendXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordUnpauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ResumeXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WfEndXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/RecoveryService.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/PollablePriorityDelayQueue.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/XCallable.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/resources/oozie-default.xml 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionInputCheckXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionStartXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestCallableQueueService.java 1209829 
> 
> Diff: https://reviews.apache.org/r/3030/diff
> 
> 
> Testing
> -------
> 
> Regression for all the Commands in addition for specific testing for the Interrupt Driven Map
> 
> 
> Thanks,
> 
> Mohamed
> 
>


Re: Review Request: OOZIE-591: Oozie continues to materialize new actions after end date modification

Posted by Alejandro Abdelnur <tu...@cloudera.com>.

> On 2012-01-06 23:24:17, Alejandro Abdelnur wrote:
> > The following is now being done in several commands:
> > 
> >   protected String getEntityKey() {
> >         if (coordAction != null) {
> >             return coordAction.getJobId();
> >         }
> >         return null;
> >   }
> > 
> > The entity cannot be NULL or lazy loaded. When the command is created the entity key MUST be provided. this guarding and returning NULL if not present is not correct as the lock is acquired before doing a loadState.
> > 
> > So, all these checks should be removed and the entity key should be provided in all XCommand constructors.
> > 
> > -------
> > 
> > There are still many false changes adding 'this.'
> > 
> > -------
> > 
> > I'm still questioning the need for adding the lockKey, the entityKey should be enough.
> > 
> > If not, please explain why
> > 
> > -------
> > 
> > Thxs.
> > 
> > Alejandro
> 
> Mohamed Battisha wrote:
>     As far as providing the entity key providers [like the coordAction in this case] in the constructor, I changed some of the XCommands to assure this is taking place. I am not sure if i did cover all the commands this is why i added this check to avoid a null pointer exception. I will eliminate these changes and will open another Jira to track this is in a deterministic way, meanwhile checking the lock while executing the XCommand instead of at the Callable Queue service will assure the entity key is initialized as all of them will be initialized prior of calling execute.
>     
>     As far as the lock key/entity key.  The entity key is part of XCommand and can not be accessed through the XCallable interface, as such I created and interface method in the XCallable interface [getLockKey()] to return the  the entity key ] which will be used while creating and maintaining the interrupt map.  This was the simplest way of doing it, other options will require a bigger footprint which I prefere to avoid at this point
>

on first paragraph, sounds good.

on second paragraph, my concern with 2 keys here is that the lock is acquired for the entity-key and the interruption is is keyed with the lock-key, and they may be different. thus creating weird issues hard to track. bottom line (IMO), the key used for locking MUST be the same as for interruption and this could not be changed.


- Alejandro


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3030/#review4229
-----------------------------------------------------------


On 2012-01-06 13:45:50, Mohamed Battisha wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/3030/
> -----------------------------------------------------------
> 
> (Updated 2012-01-06 13:45:50)
> 
> 
> Review request for oozie.
> 
> 
> Summary
> -------
> 
> Oozie continues to materialize new actions after end date modification. The main issue is related how Oozie materialized the actions based on a FIFO priority queue. Changing the status of a bundle/coordinator job should take higher priority than executing this job. 
> 
> The main idea is to enable the queue to handle interruptions. Mainly, once you finished what you are working on currently, you should focus on executing this next action. 
> 
> The issue can be illustrated as follow: 
> 
> 1. Configure a pipeline to run for 1 hour 
> 2. Start the pipeline 
> 3. After it starts materializing new actions, change the end time (in my example - to 10 minutes after the pipeline 
> starts) 
> 4. Monitor the coordinator apps - they will continue to materialize new actions past the end time.
> 
> 
> This addresses bug OOZIE-591.
>     https://issues.apache.org/jira/browse/OOZIE-591
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/Command.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/XCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePauseXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStatusUpdateXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleUnpauseXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionNotificationXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionReadyXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionStartXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionTimeOutXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdateXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPauseXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordRerunXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordUnpauseXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WfEndXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/RecoveryService.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/XCallable.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/resources/oozie-default.xml 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionInputCheckXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionStartXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestCallableQueueService.java 1228133 
> 
> Diff: https://reviews.apache.org/r/3030/diff
> 
> 
> Testing
> -------
> 
> Regression for all the Commands in addition for specific testing for the Interrupt Driven Map
> 
> 
> Thanks,
> 
> Mohamed
> 
>


Re: Review Request: OOZIE-591: Oozie continues to materialize new actions after end date modification

Posted by Mohamed Battisha <mb...@yahoo.com>.

> On 2012-01-06 23:24:17, Alejandro Abdelnur wrote:
> > The following is now being done in several commands:
> > 
> >   protected String getEntityKey() {
> >         if (coordAction != null) {
> >             return coordAction.getJobId();
> >         }
> >         return null;
> >   }
> > 
> > The entity cannot be NULL or lazy loaded. When the command is created the entity key MUST be provided. this guarding and returning NULL if not present is not correct as the lock is acquired before doing a loadState.
> > 
> > So, all these checks should be removed and the entity key should be provided in all XCommand constructors.
> > 
> > -------
> > 
> > There are still many false changes adding 'this.'
> > 
> > -------
> > 
> > I'm still questioning the need for adding the lockKey, the entityKey should be enough.
> > 
> > If not, please explain why
> > 
> > -------
> > 
> > Thxs.
> > 
> > Alejandro
> 
> Mohamed Battisha wrote:
>     As far as providing the entity key providers [like the coordAction in this case] in the constructor, I changed some of the XCommands to assure this is taking place. I am not sure if i did cover all the commands this is why i added this check to avoid a null pointer exception. I will eliminate these changes and will open another Jira to track this is in a deterministic way, meanwhile checking the lock while executing the XCommand instead of at the Callable Queue service will assure the entity key is initialized as all of them will be initialized prior of calling execute.
>     
>     As far as the lock key/entity key.  The entity key is part of XCommand and can not be accessed through the XCallable interface, as such I created and interface method in the XCallable interface [getLockKey()] to return the  the entity key ] which will be used while creating and maintaining the interrupt map.  This was the simplest way of doing it, other options will require a bigger footprint which I prefere to avoid at this point
>
> 
> Alejandro Abdelnur wrote:
>     on first paragraph, sounds good.
>     
>     on second paragraph, my concern with 2 keys here is that the lock is acquired for the entity-key and the interruption is is keyed with the lock-key, and they may be different. thus creating weird issues hard to track. bottom line (IMO), the key used for locking MUST be the same as for interruption and this could not be changed.
> 
> Mohamed Battisha wrote:
>     Agreed and this is why we are using the same key
>     
>     @line 480 of XCommand.Java
>     
>        * Return the entity Key
>          * @return the callable Lock key to be used by the interrupt driven map
>          */
>         public String getLockKey() {
>             return getEntityKey();
>         }
>
> 
> Alejandro Abdelnur wrote:
>     but they why not using getEntityKey() directly?

Unfortunately getEntityKey() can not be exposed to the XCallable interface.  It will require changing all the XCommand(s) to change the signature of the getEntityKey which will have a bigger footprint. 


- Mohamed


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3030/#review4229
-----------------------------------------------------------


On 2012-01-09 19:19:08, Mohamed Battisha wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/3030/
> -----------------------------------------------------------
> 
> (Updated 2012-01-09 19:19:08)
> 
> 
> Review request for oozie.
> 
> 
> Summary
> -------
> 
> Oozie continues to materialize new actions after end date modification. The main issue is related how Oozie materialized the actions based on a FIFO priority queue. Changing the status of a bundle/coordinator job should take higher priority than executing this job. 
> 
> The main idea is to enable the queue to handle interruptions. Mainly, once you finished what you are working on currently, you should focus on executing this next action. 
> 
> The issue can be illustrated as follow: 
> 
> 1. Configure a pipeline to run for 1 hour 
> 2. Start the pipeline 
> 3. After it starts materializing new actions, change the end time (in my example - to 10 minutes after the pipeline 
> starts) 
> 4. Monitor the coordinator apps - they will continue to materialize new actions past the end time.
> 
> 
> This addresses bug OOZIE-591.
>     https://issues.apache.org/jira/browse/OOZIE-591
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/Command.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/XCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionReadyXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionStartXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordRerunXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/RecoveryService.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/XCallable.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/resources/oozie-default.xml 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionInputCheckXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionStartXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestCallableQueueService.java 1228133 
> 
> Diff: https://reviews.apache.org/r/3030/diff
> 
> 
> Testing
> -------
> 
> Regression for all the Commands in addition for specific testing for the Interrupt Driven Map
> 
> 
> Thanks,
> 
> Mohamed
> 
>


Re: Review Request: OOZIE-591: Oozie continues to materialize new actions after end date modification

Posted by Mohamed Battisha <mb...@yahoo.com>.

> On 2012-01-06 23:24:17, Alejandro Abdelnur wrote:
> > The following is now being done in several commands:
> > 
> >   protected String getEntityKey() {
> >         if (coordAction != null) {
> >             return coordAction.getJobId();
> >         }
> >         return null;
> >   }
> > 
> > The entity cannot be NULL or lazy loaded. When the command is created the entity key MUST be provided. this guarding and returning NULL if not present is not correct as the lock is acquired before doing a loadState.
> > 
> > So, all these checks should be removed and the entity key should be provided in all XCommand constructors.
> > 
> > -------
> > 
> > There are still many false changes adding 'this.'
> > 
> > -------
> > 
> > I'm still questioning the need for adding the lockKey, the entityKey should be enough.
> > 
> > If not, please explain why
> > 
> > -------
> > 
> > Thxs.
> > 
> > Alejandro

As far as providing the entity key providers [like the coordAction in this case] in the constructor, I changed some of the XCommands to assure this is taking place. I am not sure if i did cover all the commands this is why i added this check to avoid a null pointer exception. I will eliminate these changes and will open another Jira to track this is in a deterministic way, meanwhile checking the lock while executing the XCommand instead of at the Callable Queue service will assure the entity key is initialized as all of them will be initialized prior of calling execute.

As far as the lock key/entity key.  The entity key is part of XCommand and can not be accessed through the XCallable interface, as such I created and interface method in the XCallable interface [getLockKey()] to return the  the entity key ] which will be used while creating and maintaining the interrupt map.  This was the simplest way of doing it, other options will require a bigger footprint which I prefere to avoid at this point


- Mohamed


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3030/#review4229
-----------------------------------------------------------


On 2012-01-06 13:45:50, Mohamed Battisha wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/3030/
> -----------------------------------------------------------
> 
> (Updated 2012-01-06 13:45:50)
> 
> 
> Review request for oozie.
> 
> 
> Summary
> -------
> 
> Oozie continues to materialize new actions after end date modification. The main issue is related how Oozie materialized the actions based on a FIFO priority queue. Changing the status of a bundle/coordinator job should take higher priority than executing this job. 
> 
> The main idea is to enable the queue to handle interruptions. Mainly, once you finished what you are working on currently, you should focus on executing this next action. 
> 
> The issue can be illustrated as follow: 
> 
> 1. Configure a pipeline to run for 1 hour 
> 2. Start the pipeline 
> 3. After it starts materializing new actions, change the end time (in my example - to 10 minutes after the pipeline 
> starts) 
> 4. Monitor the coordinator apps - they will continue to materialize new actions past the end time.
> 
> 
> This addresses bug OOZIE-591.
>     https://issues.apache.org/jira/browse/OOZIE-591
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/Command.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/XCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePauseXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStatusUpdateXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleUnpauseXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionNotificationXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionReadyXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionStartXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionTimeOutXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdateXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPauseXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordRerunXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordUnpauseXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WfEndXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/RecoveryService.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/XCallable.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/resources/oozie-default.xml 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionInputCheckXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionStartXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestCallableQueueService.java 1228133 
> 
> Diff: https://reviews.apache.org/r/3030/diff
> 
> 
> Testing
> -------
> 
> Regression for all the Commands in addition for specific testing for the Interrupt Driven Map
> 
> 
> Thanks,
> 
> Mohamed
> 
>


Re: Review Request: OOZIE-591: Oozie continues to materialize new actions after end date modification

Posted by Alejandro Abdelnur <tu...@cloudera.com>.

> On 2012-01-06 23:24:17, Alejandro Abdelnur wrote:
> > The following is now being done in several commands:
> > 
> >   protected String getEntityKey() {
> >         if (coordAction != null) {
> >             return coordAction.getJobId();
> >         }
> >         return null;
> >   }
> > 
> > The entity cannot be NULL or lazy loaded. When the command is created the entity key MUST be provided. this guarding and returning NULL if not present is not correct as the lock is acquired before doing a loadState.
> > 
> > So, all these checks should be removed and the entity key should be provided in all XCommand constructors.
> > 
> > -------
> > 
> > There are still many false changes adding 'this.'
> > 
> > -------
> > 
> > I'm still questioning the need for adding the lockKey, the entityKey should be enough.
> > 
> > If not, please explain why
> > 
> > -------
> > 
> > Thxs.
> > 
> > Alejandro
> 
> Mohamed Battisha wrote:
>     As far as providing the entity key providers [like the coordAction in this case] in the constructor, I changed some of the XCommands to assure this is taking place. I am not sure if i did cover all the commands this is why i added this check to avoid a null pointer exception. I will eliminate these changes and will open another Jira to track this is in a deterministic way, meanwhile checking the lock while executing the XCommand instead of at the Callable Queue service will assure the entity key is initialized as all of them will be initialized prior of calling execute.
>     
>     As far as the lock key/entity key.  The entity key is part of XCommand and can not be accessed through the XCallable interface, as such I created and interface method in the XCallable interface [getLockKey()] to return the  the entity key ] which will be used while creating and maintaining the interrupt map.  This was the simplest way of doing it, other options will require a bigger footprint which I prefere to avoid at this point
>
> 
> Alejandro Abdelnur wrote:
>     on first paragraph, sounds good.
>     
>     on second paragraph, my concern with 2 keys here is that the lock is acquired for the entity-key and the interruption is is keyed with the lock-key, and they may be different. thus creating weird issues hard to track. bottom line (IMO), the key used for locking MUST be the same as for interruption and this could not be changed.
> 
> Mohamed Battisha wrote:
>     Agreed and this is why we are using the same key
>     
>     @line 480 of XCommand.Java
>     
>        * Return the entity Key
>          * @return the callable Lock key to be used by the interrupt driven map
>          */
>         public String getLockKey() {
>             return getEntityKey();
>         }
>

but they why not using getEntityKey() directly?


- Alejandro


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3030/#review4229
-----------------------------------------------------------


On 2012-01-09 19:19:08, Mohamed Battisha wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/3030/
> -----------------------------------------------------------
> 
> (Updated 2012-01-09 19:19:08)
> 
> 
> Review request for oozie.
> 
> 
> Summary
> -------
> 
> Oozie continues to materialize new actions after end date modification. The main issue is related how Oozie materialized the actions based on a FIFO priority queue. Changing the status of a bundle/coordinator job should take higher priority than executing this job. 
> 
> The main idea is to enable the queue to handle interruptions. Mainly, once you finished what you are working on currently, you should focus on executing this next action. 
> 
> The issue can be illustrated as follow: 
> 
> 1. Configure a pipeline to run for 1 hour 
> 2. Start the pipeline 
> 3. After it starts materializing new actions, change the end time (in my example - to 10 minutes after the pipeline 
> starts) 
> 4. Monitor the coordinator apps - they will continue to materialize new actions past the end time.
> 
> 
> This addresses bug OOZIE-591.
>     https://issues.apache.org/jira/browse/OOZIE-591
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/Command.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/XCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionReadyXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionStartXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordRerunXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/RecoveryService.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/XCallable.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/resources/oozie-default.xml 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionInputCheckXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionStartXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestCallableQueueService.java 1228133 
> 
> Diff: https://reviews.apache.org/r/3030/diff
> 
> 
> Testing
> -------
> 
> Regression for all the Commands in addition for specific testing for the Interrupt Driven Map
> 
> 
> Thanks,
> 
> Mohamed
> 
>


Re: Review Request: OOZIE-591: Oozie continues to materialize new actions after end date modification

Posted by Mohamed Battisha <mb...@yahoo.com>.

> On 2012-01-06 23:24:17, Alejandro Abdelnur wrote:
> > The following is now being done in several commands:
> > 
> >   protected String getEntityKey() {
> >         if (coordAction != null) {
> >             return coordAction.getJobId();
> >         }
> >         return null;
> >   }
> > 
> > The entity cannot be NULL or lazy loaded. When the command is created the entity key MUST be provided. this guarding and returning NULL if not present is not correct as the lock is acquired before doing a loadState.
> > 
> > So, all these checks should be removed and the entity key should be provided in all XCommand constructors.
> > 
> > -------
> > 
> > There are still many false changes adding 'this.'
> > 
> > -------
> > 
> > I'm still questioning the need for adding the lockKey, the entityKey should be enough.
> > 
> > If not, please explain why
> > 
> > -------
> > 
> > Thxs.
> > 
> > Alejandro
> 
> Mohamed Battisha wrote:
>     As far as providing the entity key providers [like the coordAction in this case] in the constructor, I changed some of the XCommands to assure this is taking place. I am not sure if i did cover all the commands this is why i added this check to avoid a null pointer exception. I will eliminate these changes and will open another Jira to track this is in a deterministic way, meanwhile checking the lock while executing the XCommand instead of at the Callable Queue service will assure the entity key is initialized as all of them will be initialized prior of calling execute.
>     
>     As far as the lock key/entity key.  The entity key is part of XCommand and can not be accessed through the XCallable interface, as such I created and interface method in the XCallable interface [getLockKey()] to return the  the entity key ] which will be used while creating and maintaining the interrupt map.  This was the simplest way of doing it, other options will require a bigger footprint which I prefere to avoid at this point
>
> 
> Alejandro Abdelnur wrote:
>     on first paragraph, sounds good.
>     
>     on second paragraph, my concern with 2 keys here is that the lock is acquired for the entity-key and the interruption is is keyed with the lock-key, and they may be different. thus creating weird issues hard to track. bottom line (IMO), the key used for locking MUST be the same as for interruption and this could not be changed.

Agreed and this is why we are using the same key

@line 480 of XCommand.Java

   * Return the entity Key
     * @return the callable Lock key to be used by the interrupt driven map
     */
    public String getLockKey() {
        return getEntityKey();
    }


- Mohamed


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3030/#review4229
-----------------------------------------------------------


On 2012-01-09 19:19:08, Mohamed Battisha wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/3030/
> -----------------------------------------------------------
> 
> (Updated 2012-01-09 19:19:08)
> 
> 
> Review request for oozie.
> 
> 
> Summary
> -------
> 
> Oozie continues to materialize new actions after end date modification. The main issue is related how Oozie materialized the actions based on a FIFO priority queue. Changing the status of a bundle/coordinator job should take higher priority than executing this job. 
> 
> The main idea is to enable the queue to handle interruptions. Mainly, once you finished what you are working on currently, you should focus on executing this next action. 
> 
> The issue can be illustrated as follow: 
> 
> 1. Configure a pipeline to run for 1 hour 
> 2. Start the pipeline 
> 3. After it starts materializing new actions, change the end time (in my example - to 10 minutes after the pipeline 
> starts) 
> 4. Monitor the coordinator apps - they will continue to materialize new actions past the end time.
> 
> 
> This addresses bug OOZIE-591.
>     https://issues.apache.org/jira/browse/OOZIE-591
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/Command.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/XCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionReadyXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionStartXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordRerunXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/RecoveryService.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/XCallable.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/resources/oozie-default.xml 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionInputCheckXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionStartXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestCallableQueueService.java 1228133 
> 
> Diff: https://reviews.apache.org/r/3030/diff
> 
> 
> Testing
> -------
> 
> Regression for all the Commands in addition for specific testing for the Interrupt Driven Map
> 
> 
> Thanks,
> 
> Mohamed
> 
>


Re: Review Request: OOZIE-591: Oozie continues to materialize new actions after end date modification

Posted by Alejandro Abdelnur <tu...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3030/#review4229
-----------------------------------------------------------


The following is now being done in several commands:

  protected String getEntityKey() {
        if (coordAction != null) {
            return coordAction.getJobId();
        }
        return null;
  }

The entity cannot be NULL or lazy loaded. When the command is created the entity key MUST be provided. this guarding and returning NULL if not present is not correct as the lock is acquired before doing a loadState.

So, all these checks should be removed and the entity key should be provided in all XCommand constructors.

-------

There are still many false changes adding 'this.'

-------

I'm still questioning the need for adding the lockKey, the entityKey should be enough.

If not, please explain why

-------

Thxs.

Alejandro

- Alejandro


On 2012-01-06 13:45:50, Mohamed Battisha wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/3030/
> -----------------------------------------------------------
> 
> (Updated 2012-01-06 13:45:50)
> 
> 
> Review request for oozie.
> 
> 
> Summary
> -------
> 
> Oozie continues to materialize new actions after end date modification. The main issue is related how Oozie materialized the actions based on a FIFO priority queue. Changing the status of a bundle/coordinator job should take higher priority than executing this job. 
> 
> The main idea is to enable the queue to handle interruptions. Mainly, once you finished what you are working on currently, you should focus on executing this next action. 
> 
> The issue can be illustrated as follow: 
> 
> 1. Configure a pipeline to run for 1 hour 
> 2. Start the pipeline 
> 3. After it starts materializing new actions, change the end time (in my example - to 10 minutes after the pipeline 
> starts) 
> 4. Monitor the coordinator apps - they will continue to materialize new actions past the end time.
> 
> 
> This addresses bug OOZIE-591.
>     https://issues.apache.org/jira/browse/OOZIE-591
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/Command.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/XCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePauseXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStatusUpdateXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleUnpauseXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionNotificationXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionReadyXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionStartXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionTimeOutXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdateXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPauseXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordRerunXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordUnpauseXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WfEndXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/RecoveryService.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/XCallable.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/resources/oozie-default.xml 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionInputCheckXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionStartXCommand.java 1228133 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestCallableQueueService.java 1228133 
> 
> Diff: https://reviews.apache.org/r/3030/diff
> 
> 
> Testing
> -------
> 
> Regression for all the Commands in addition for specific testing for the Interrupt Driven Map
> 
> 
> Thanks,
> 
> Mohamed
> 
>


Re: Review Request: OOZIE-591: Oozie continues to materialize new actions after end date modification

Posted by Alejandro Abdelnur <tu...@cloudera.com>.

> On 2012-01-12 20:10:13, Alejandro Abdelnur wrote:
> > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java, line 692
> > <https://reviews.apache.org/r/3030/diff/7-8/?file=68031#file68031line692>
> >
> >     this synchronized is not needed as the map is a concurrenthashmap
> 
> Mohamed Battisha wrote:
>     The main issue is not with the map as it is concurrent map as you mentioned
>     The issue is with the list, The race condition i am trying to avoid is to add an element to the list while removing it from the map.
>     
>     Synchronizing across the block and while removing the list from the map will take care of the race condition

adding an element to the list while removing it from the map is fine, it means a command got lucky and will be processed as part of an interruption that is about to start, that will create any inconcistencies.


- Alejandro


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3030/#review4328
-----------------------------------------------------------


On 2012-01-12 03:04:27, Mohamed Battisha wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/3030/
> -----------------------------------------------------------
> 
> (Updated 2012-01-12 03:04:27)
> 
> 
> Review request for oozie.
> 
> 
> Summary
> -------
> 
> Oozie continues to materialize new actions after end date modification. The main issue is related how Oozie materialized the actions based on a FIFO priority queue. Changing the status of a bundle/coordinator job should take higher priority than executing this job. 
> 
> The main idea is to enable the queue to handle interruptions. Mainly, once you finished what you are working on currently, you should focus on executing this next action. 
> 
> The issue can be illustrated as follow: 
> 
> 1. Configure a pipeline to run for 1 hour 
> 2. Start the pipeline 
> 3. After it starts materializing new actions, change the end time (in my example - to 10 minutes after the pipeline 
> starts) 
> 4. Monitor the coordinator apps - they will continue to materialize new actions past the end time.
> 
> 
> This addresses bug OOZIE-591.
>     https://issues.apache.org/jira/browse/OOZIE-591
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/Command.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/XCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobChangeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobResumeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobSuspendXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobsXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleKillXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePauseXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePurgeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleRerunXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStartXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStatusUpdateXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleSubmitXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleUnpauseXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionCheckXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInfoXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionNotificationXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionReadyXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionStartXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionTimeOutXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdateXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordChangeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordJobXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordJobsXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordKillXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPauseXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPurgeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordRerunXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordResumeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSubmitXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSuspendXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordUnpauseXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/SLAEventsXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionCheckXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionEndXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionKillXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionStartXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/CompletedActionXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/DefinitionXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ExternalIdXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/JobXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/JobsXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/KillXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/NotificationXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/PurgeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ReRunXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ResumeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SignalXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SubmitMRXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SubmitPigXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SubmitXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SuspendXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WfEndXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WorkflowActionInfoXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/RecoveryService.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/XCallable.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/resources/oozie-default.xml 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionInputCheckXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionStartXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestCallableQueueService.java 1230165 
> 
> Diff: https://reviews.apache.org/r/3030/diff
> 
> 
> Testing
> -------
> 
> Regression for all the Commands in addition for specific testing for the Interrupt Driven Map
> 
> 
> Thanks,
> 
> Mohamed
> 
>


Re: Review Request: OOZIE-591: Oozie continues to materialize new actions after end date modification

Posted by Mohamed Battisha <mb...@yahoo.com>.

> On 2012-01-12 20:10:13, Alejandro Abdelnur wrote:
> > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java, line 692
> > <https://reviews.apache.org/r/3030/diff/7-8/?file=68031#file68031line692>
> >
> >     this synchronized is not needed as the map is a concurrenthashmap

The main issue is not with the map as it is concurrent map as you mentioned
The issue is with the list, The race condition i am trying to avoid is to add an element to the list while removing it from the map.

Synchronizing across the block and while removing the list from the map will take care of the race condition


- Mohamed


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3030/#review4328
-----------------------------------------------------------


On 2012-01-12 03:04:27, Mohamed Battisha wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/3030/
> -----------------------------------------------------------
> 
> (Updated 2012-01-12 03:04:27)
> 
> 
> Review request for oozie.
> 
> 
> Summary
> -------
> 
> Oozie continues to materialize new actions after end date modification. The main issue is related how Oozie materialized the actions based on a FIFO priority queue. Changing the status of a bundle/coordinator job should take higher priority than executing this job. 
> 
> The main idea is to enable the queue to handle interruptions. Mainly, once you finished what you are working on currently, you should focus on executing this next action. 
> 
> The issue can be illustrated as follow: 
> 
> 1. Configure a pipeline to run for 1 hour 
> 2. Start the pipeline 
> 3. After it starts materializing new actions, change the end time (in my example - to 10 minutes after the pipeline 
> starts) 
> 4. Monitor the coordinator apps - they will continue to materialize new actions past the end time.
> 
> 
> This addresses bug OOZIE-591.
>     https://issues.apache.org/jira/browse/OOZIE-591
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/Command.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/XCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobChangeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobResumeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobSuspendXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobsXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleKillXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePauseXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePurgeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleRerunXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStartXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStatusUpdateXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleSubmitXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleUnpauseXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionCheckXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInfoXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionNotificationXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionReadyXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionStartXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionTimeOutXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdateXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordChangeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordJobXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordJobsXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordKillXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPauseXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPurgeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordRerunXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordResumeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSubmitXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSuspendXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordUnpauseXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/SLAEventsXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionCheckXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionEndXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionKillXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionStartXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/CompletedActionXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/DefinitionXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ExternalIdXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/JobXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/JobsXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/KillXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/NotificationXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/PurgeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ReRunXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ResumeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SignalXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SubmitMRXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SubmitPigXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SubmitXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SuspendXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WfEndXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WorkflowActionInfoXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/RecoveryService.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/XCallable.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/resources/oozie-default.xml 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionInputCheckXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionStartXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestCallableQueueService.java 1230165 
> 
> Diff: https://reviews.apache.org/r/3030/diff
> 
> 
> Testing
> -------
> 
> Regression for all the Commands in addition for specific testing for the Interrupt Driven Map
> 
> 
> Thanks,
> 
> Mohamed
> 
>


Re: Review Request: OOZIE-591: Oozie continues to materialize new actions after end date modification

Posted by Alejandro Abdelnur <tu...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3030/#review4328
-----------------------------------------------------------



http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java
<https://reviews.apache.org/r/3030/#comment9737>

    this synchronized is not needed as the map is a concurrenthashmap



http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java
<https://reviews.apache.org/r/3030/#comment9738>

    'new ArrayList<...>' should be wrapped with Collections.synchronizedList(...)


- Alejandro


On 2012-01-12 03:04:27, Mohamed Battisha wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/3030/
> -----------------------------------------------------------
> 
> (Updated 2012-01-12 03:04:27)
> 
> 
> Review request for oozie.
> 
> 
> Summary
> -------
> 
> Oozie continues to materialize new actions after end date modification. The main issue is related how Oozie materialized the actions based on a FIFO priority queue. Changing the status of a bundle/coordinator job should take higher priority than executing this job. 
> 
> The main idea is to enable the queue to handle interruptions. Mainly, once you finished what you are working on currently, you should focus on executing this next action. 
> 
> The issue can be illustrated as follow: 
> 
> 1. Configure a pipeline to run for 1 hour 
> 2. Start the pipeline 
> 3. After it starts materializing new actions, change the end time (in my example - to 10 minutes after the pipeline 
> starts) 
> 4. Monitor the coordinator apps - they will continue to materialize new actions past the end time.
> 
> 
> This addresses bug OOZIE-591.
>     https://issues.apache.org/jira/browse/OOZIE-591
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/Command.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/XCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobChangeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobResumeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobSuspendXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobsXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleKillXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePauseXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePurgeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleRerunXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStartXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStatusUpdateXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleSubmitXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleUnpauseXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionCheckXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInfoXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionNotificationXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionReadyXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionStartXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionTimeOutXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdateXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordChangeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordJobXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordJobsXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordKillXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPauseXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPurgeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordRerunXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordResumeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSubmitXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSuspendXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordUnpauseXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/SLAEventsXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionCheckXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionEndXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionKillXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionStartXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/CompletedActionXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/DefinitionXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ExternalIdXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/JobXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/JobsXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/KillXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/NotificationXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/PurgeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ReRunXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ResumeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SignalXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SubmitMRXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SubmitPigXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SubmitXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SuspendXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WfEndXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WorkflowActionInfoXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/RecoveryService.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/XCallable.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/resources/oozie-default.xml 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionInputCheckXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionStartXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestCallableQueueService.java 1230165 
> 
> Diff: https://reviews.apache.org/r/3030/diff
> 
> 
> Testing
> -------
> 
> Regression for all the Commands in addition for specific testing for the Interrupt Driven Map
> 
> 
> Thanks,
> 
> Mohamed
> 
>


Re: Review Request: OOZIE-591: Oozie continues to materialize new actions after end date modification

Posted by Mohamed Battisha <mb...@yahoo.com>.

> On 2012-01-12 22:48:03, Alejandro Abdelnur wrote:
> > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java, line 727
> > <https://reviews.apache.org/r/3030/diff/8-9/?file=68276#file68276line727>
> >
> >     lines 724/725should be:
> >     
> >     ----
> >     List<XCallable<?>> newList = Collections.synchronizedList(new ArrayList<XCallable<?>>());
> >     List<XCallable<?>> interruptList = interruptCommandsMap.putIfAbsent(callable.getEntityKey(), newList);
> >     if (interruptList == null) {
> >       interruptList = newList;
> >     }
> >     ----
> 
> Mohamed Battisha wrote:
>     Is there any reason you believe putIfAbsent will return a null value.
>     
>     in any condition it should not. 
>     remember the hash map will never allow a null key or value to be inserted in
>     and if the key is absent, a new list will be created for us
>     
>     I believe the other code would work fine as we are wrapping the list in a synchronized collection which will assure a serial add
>
> 
> Alejandro Abdelnur wrote:
>     Per javadocs:
>     
>     ----
>     Returns:
>     the previous value associated with the specified key, or null if there was no mapping for the key. (A null return can also indicate that the map previously associated null with the key, if the implementation supports null values.)
>     ----
>     
>     HashMap allows NULLs, Hashtable doesn't
>     
>     
>     Your current code does not add the synchronized list to the map, instead synchronizes the current reference in scope, thus 2 concurrent threads will have different synchronized wrappers and synchronization will be not enforced on the underlying list.
>     
>     
>

My mistake. You are right, will take care of it 


- Mohamed


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3030/#review4340
-----------------------------------------------------------


On 2012-01-12 21:36:09, Mohamed Battisha wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/3030/
> -----------------------------------------------------------
> 
> (Updated 2012-01-12 21:36:09)
> 
> 
> Review request for oozie.
> 
> 
> Summary
> -------
> 
> Oozie continues to materialize new actions after end date modification. The main issue is related how Oozie materialized the actions based on a FIFO priority queue. Changing the status of a bundle/coordinator job should take higher priority than executing this job. 
> 
> The main idea is to enable the queue to handle interruptions. Mainly, once you finished what you are working on currently, you should focus on executing this next action. 
> 
> The issue can be illustrated as follow: 
> 
> 1. Configure a pipeline to run for 1 hour 
> 2. Start the pipeline 
> 3. After it starts materializing new actions, change the end time (in my example - to 10 minutes after the pipeline 
> starts) 
> 4. Monitor the coordinator apps - they will continue to materialize new actions past the end time.
> 
> 
> This addresses bug OOZIE-591.
>     https://issues.apache.org/jira/browse/OOZIE-591
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/Command.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/XCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobChangeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobResumeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobSuspendXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobsXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleKillXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePauseXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePurgeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleRerunXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStartXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStatusUpdateXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleSubmitXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleUnpauseXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionCheckXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInfoXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionNotificationXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionReadyXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionStartXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionTimeOutXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdateXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordChangeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordJobXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordJobsXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordKillXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPauseXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPurgeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordRerunXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordResumeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSubmitXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSuspendXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordUnpauseXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/SLAEventsXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionCheckXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionEndXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionKillXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionStartXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/CompletedActionXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/DefinitionXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ExternalIdXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/JobXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/JobsXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/KillXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/NotificationXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/PurgeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ReRunXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ResumeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SignalXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SubmitMRXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SubmitPigXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SubmitXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SuspendXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WfEndXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WorkflowActionInfoXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/RecoveryService.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/XCallable.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/resources/oozie-default.xml 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionInputCheckXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionStartXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestCallableQueueService.java 1230165 
> 
> Diff: https://reviews.apache.org/r/3030/diff
> 
> 
> Testing
> -------
> 
> Regression for all the Commands in addition for specific testing for the Interrupt Driven Map
> 
> 
> Thanks,
> 
> Mohamed
> 
>


Re: Review Request: OOZIE-591: Oozie continues to materialize new actions after end date modification

Posted by Mohamed Battisha <mb...@yahoo.com>.

> On 2012-01-12 22:48:03, Alejandro Abdelnur wrote:
> > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java, line 727
> > <https://reviews.apache.org/r/3030/diff/8-9/?file=68276#file68276line727>
> >
> >     lines 724/725should be:
> >     
> >     ----
> >     List<XCallable<?>> newList = Collections.synchronizedList(new ArrayList<XCallable<?>>());
> >     List<XCallable<?>> interruptList = interruptCommandsMap.putIfAbsent(callable.getEntityKey(), newList);
> >     if (interruptList == null) {
> >       interruptList = newList;
> >     }
> >     ----

Is there any reason you believe putIfAbsent will return a null value.

in any condition it should not. 
remember the hash map will never allow a null key or value to be inserted in
and if the key is absent, a new list will be created for us

I believe the other code would work fine as we are wrapping the list in a synchronized collection which will assure a serial add


- Mohamed


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3030/#review4340
-----------------------------------------------------------


On 2012-01-12 21:36:09, Mohamed Battisha wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/3030/
> -----------------------------------------------------------
> 
> (Updated 2012-01-12 21:36:09)
> 
> 
> Review request for oozie.
> 
> 
> Summary
> -------
> 
> Oozie continues to materialize new actions after end date modification. The main issue is related how Oozie materialized the actions based on a FIFO priority queue. Changing the status of a bundle/coordinator job should take higher priority than executing this job. 
> 
> The main idea is to enable the queue to handle interruptions. Mainly, once you finished what you are working on currently, you should focus on executing this next action. 
> 
> The issue can be illustrated as follow: 
> 
> 1. Configure a pipeline to run for 1 hour 
> 2. Start the pipeline 
> 3. After it starts materializing new actions, change the end time (in my example - to 10 minutes after the pipeline 
> starts) 
> 4. Monitor the coordinator apps - they will continue to materialize new actions past the end time.
> 
> 
> This addresses bug OOZIE-591.
>     https://issues.apache.org/jira/browse/OOZIE-591
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/Command.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/XCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobChangeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobResumeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobSuspendXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobsXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleKillXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePauseXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePurgeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleRerunXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStartXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStatusUpdateXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleSubmitXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleUnpauseXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionCheckXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInfoXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionNotificationXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionReadyXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionStartXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionTimeOutXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdateXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordChangeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordJobXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordJobsXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordKillXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPauseXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPurgeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordRerunXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordResumeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSubmitXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSuspendXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordUnpauseXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/SLAEventsXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionCheckXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionEndXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionKillXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionStartXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/CompletedActionXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/DefinitionXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ExternalIdXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/JobXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/JobsXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/KillXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/NotificationXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/PurgeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ReRunXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ResumeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SignalXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SubmitMRXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SubmitPigXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SubmitXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SuspendXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WfEndXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WorkflowActionInfoXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/RecoveryService.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/XCallable.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/resources/oozie-default.xml 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionInputCheckXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionStartXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestCallableQueueService.java 1230165 
> 
> Diff: https://reviews.apache.org/r/3030/diff
> 
> 
> Testing
> -------
> 
> Regression for all the Commands in addition for specific testing for the Interrupt Driven Map
> 
> 
> Thanks,
> 
> Mohamed
> 
>


Re: Review Request: OOZIE-591: Oozie continues to materialize new actions after end date modification

Posted by Alejandro Abdelnur <tu...@cloudera.com>.

> On 2012-01-12 22:48:03, Alejandro Abdelnur wrote:
> > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java, line 727
> > <https://reviews.apache.org/r/3030/diff/8-9/?file=68276#file68276line727>
> >
> >     lines 724/725should be:
> >     
> >     ----
> >     List<XCallable<?>> newList = Collections.synchronizedList(new ArrayList<XCallable<?>>());
> >     List<XCallable<?>> interruptList = interruptCommandsMap.putIfAbsent(callable.getEntityKey(), newList);
> >     if (interruptList == null) {
> >       interruptList = newList;
> >     }
> >     ----
> 
> Mohamed Battisha wrote:
>     Is there any reason you believe putIfAbsent will return a null value.
>     
>     in any condition it should not. 
>     remember the hash map will never allow a null key or value to be inserted in
>     and if the key is absent, a new list will be created for us
>     
>     I believe the other code would work fine as we are wrapping the list in a synchronized collection which will assure a serial add
>

Per javadocs:

----
Returns:
the previous value associated with the specified key, or null if there was no mapping for the key. (A null return can also indicate that the map previously associated null with the key, if the implementation supports null values.)
----

HashMap allows NULLs, Hashtable doesn't


Your current code does not add the synchronized list to the map, instead synchronizes the current reference in scope, thus 2 concurrent threads will have different synchronized wrappers and synchronization will be not enforced on the underlying list.


- Alejandro


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3030/#review4340
-----------------------------------------------------------


On 2012-01-12 21:36:09, Mohamed Battisha wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/3030/
> -----------------------------------------------------------
> 
> (Updated 2012-01-12 21:36:09)
> 
> 
> Review request for oozie.
> 
> 
> Summary
> -------
> 
> Oozie continues to materialize new actions after end date modification. The main issue is related how Oozie materialized the actions based on a FIFO priority queue. Changing the status of a bundle/coordinator job should take higher priority than executing this job. 
> 
> The main idea is to enable the queue to handle interruptions. Mainly, once you finished what you are working on currently, you should focus on executing this next action. 
> 
> The issue can be illustrated as follow: 
> 
> 1. Configure a pipeline to run for 1 hour 
> 2. Start the pipeline 
> 3. After it starts materializing new actions, change the end time (in my example - to 10 minutes after the pipeline 
> starts) 
> 4. Monitor the coordinator apps - they will continue to materialize new actions past the end time.
> 
> 
> This addresses bug OOZIE-591.
>     https://issues.apache.org/jira/browse/OOZIE-591
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/Command.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/XCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobChangeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobResumeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobSuspendXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobsXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleKillXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePauseXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePurgeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleRerunXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStartXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStatusUpdateXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleSubmitXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleUnpauseXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionCheckXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInfoXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionNotificationXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionReadyXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionStartXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionTimeOutXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdateXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordChangeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordJobXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordJobsXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordKillXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPauseXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPurgeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordRerunXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordResumeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSubmitXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSuspendXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordUnpauseXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/SLAEventsXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionCheckXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionEndXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionKillXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionStartXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/CompletedActionXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/DefinitionXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ExternalIdXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/JobXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/JobsXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/KillXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/NotificationXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/PurgeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ReRunXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ResumeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SignalXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SubmitMRXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SubmitPigXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SubmitXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SuspendXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WfEndXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WorkflowActionInfoXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/RecoveryService.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/XCallable.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/resources/oozie-default.xml 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionInputCheckXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionStartXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestCallableQueueService.java 1230165 
> 
> Diff: https://reviews.apache.org/r/3030/diff
> 
> 
> Testing
> -------
> 
> Regression for all the Commands in addition for specific testing for the Interrupt Driven Map
> 
> 
> Thanks,
> 
> Mohamed
> 
>


Re: Review Request: OOZIE-591: Oozie continues to materialize new actions after end date modification

Posted by Alejandro Abdelnur <tu...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3030/#review4340
-----------------------------------------------------------



http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java
<https://reviews.apache.org/r/3030/#comment9771>

    lines 724/725should be:
    
    ----
    List<XCallable<?>> newList = Collections.synchronizedList(new ArrayList<XCallable<?>>());
    List<XCallable<?>> interruptList = interruptCommandsMap.putIfAbsent(callable.getEntityKey(), newList);
    if (interruptList == null) {
      interruptList = newList;
    }
    ----


- Alejandro


On 2012-01-12 21:36:09, Mohamed Battisha wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/3030/
> -----------------------------------------------------------
> 
> (Updated 2012-01-12 21:36:09)
> 
> 
> Review request for oozie.
> 
> 
> Summary
> -------
> 
> Oozie continues to materialize new actions after end date modification. The main issue is related how Oozie materialized the actions based on a FIFO priority queue. Changing the status of a bundle/coordinator job should take higher priority than executing this job. 
> 
> The main idea is to enable the queue to handle interruptions. Mainly, once you finished what you are working on currently, you should focus on executing this next action. 
> 
> The issue can be illustrated as follow: 
> 
> 1. Configure a pipeline to run for 1 hour 
> 2. Start the pipeline 
> 3. After it starts materializing new actions, change the end time (in my example - to 10 minutes after the pipeline 
> starts) 
> 4. Monitor the coordinator apps - they will continue to materialize new actions past the end time.
> 
> 
> This addresses bug OOZIE-591.
>     https://issues.apache.org/jira/browse/OOZIE-591
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/Command.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/XCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobChangeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobResumeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobSuspendXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobsXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleKillXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePauseXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePurgeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleRerunXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStartXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStatusUpdateXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleSubmitXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleUnpauseXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionCheckXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInfoXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionNotificationXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionReadyXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionStartXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionTimeOutXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdateXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordChangeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordJobXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordJobsXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordKillXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPauseXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPurgeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordRerunXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordResumeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSubmitXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSuspendXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordUnpauseXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/SLAEventsXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionCheckXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionEndXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionKillXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionStartXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/CompletedActionXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/DefinitionXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ExternalIdXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/JobXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/JobsXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/KillXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/NotificationXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/PurgeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ReRunXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ResumeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SignalXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SubmitMRXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SubmitPigXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SubmitXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SuspendXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WfEndXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WorkflowActionInfoXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/RecoveryService.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/XCallable.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/resources/oozie-default.xml 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionInputCheckXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionStartXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestCallableQueueService.java 1230165 
> 
> Diff: https://reviews.apache.org/r/3030/diff
> 
> 
> Testing
> -------
> 
> Regression for all the Commands in addition for specific testing for the Interrupt Driven Map
> 
> 
> Thanks,
> 
> Mohamed
> 
>


Re: Review Request: OOZIE-591: Oozie continues to materialize new actions after end date modification

Posted by Alejandro Abdelnur <tu...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3030/#review4347
-----------------------------------------------------------



http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java
<https://reviews.apache.org/r/3030/#comment9790>

    newList should be used as the value in the putIfAbsent, else the first callable will be added to a list that goes orphan 


- Alejandro


On 2012-01-13 01:01:22, Mohamed Battisha wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/3030/
> -----------------------------------------------------------
> 
> (Updated 2012-01-13 01:01:22)
> 
> 
> Review request for oozie.
> 
> 
> Summary
> -------
> 
> Oozie continues to materialize new actions after end date modification. The main issue is related how Oozie materialized the actions based on a FIFO priority queue. Changing the status of a bundle/coordinator job should take higher priority than executing this job. 
> 
> The main idea is to enable the queue to handle interruptions. Mainly, once you finished what you are working on currently, you should focus on executing this next action. 
> 
> The issue can be illustrated as follow: 
> 
> 1. Configure a pipeline to run for 1 hour 
> 2. Start the pipeline 
> 3. After it starts materializing new actions, change the end time (in my example - to 10 minutes after the pipeline 
> starts) 
> 4. Monitor the coordinator apps - they will continue to materialize new actions past the end time.
> 
> 
> This addresses bug OOZIE-591.
>     https://issues.apache.org/jira/browse/OOZIE-591
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/Command.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/XCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobChangeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobResumeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobSuspendXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobsXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleKillXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePauseXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePurgeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleRerunXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStartXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStatusUpdateXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleSubmitXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleUnpauseXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionCheckXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInfoXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionNotificationXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionReadyXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionStartXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionTimeOutXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdateXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordChangeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordJobXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordJobsXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordKillXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPauseXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPurgeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordRerunXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordResumeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSubmitXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSuspendXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordUnpauseXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/SLAEventsXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionCheckXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionEndXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionKillXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionStartXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/CompletedActionXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/DefinitionXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ExternalIdXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/JobXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/JobsXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/KillXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/NotificationXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/PurgeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ReRunXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ResumeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SignalXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SubmitMRXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SubmitPigXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SubmitXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SuspendXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WfEndXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WorkflowActionInfoXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/RecoveryService.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/XCallable.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/resources/oozie-default.xml 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionInputCheckXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionStartXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestCallableQueueService.java 1230165 
> 
> Diff: https://reviews.apache.org/r/3030/diff
> 
> 
> Testing
> -------
> 
> Regression for all the Commands in addition for specific testing for the Interrupt Driven Map
> 
> 
> Thanks,
> 
> Mohamed
> 
>


Re: Review Request: OOZIE-591: Oozie continues to materialize new actions after end date modification

Posted by Alejandro Abdelnur <tu...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3030/#review4349
-----------------------------------------------------------

Ship it!


- Alejandro


On 2012-01-13 01:18:34, Mohamed Battisha wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/3030/
> -----------------------------------------------------------
> 
> (Updated 2012-01-13 01:18:34)
> 
> 
> Review request for oozie.
> 
> 
> Summary
> -------
> 
> Oozie continues to materialize new actions after end date modification. The main issue is related how Oozie materialized the actions based on a FIFO priority queue. Changing the status of a bundle/coordinator job should take higher priority than executing this job. 
> 
> The main idea is to enable the queue to handle interruptions. Mainly, once you finished what you are working on currently, you should focus on executing this next action. 
> 
> The issue can be illustrated as follow: 
> 
> 1. Configure a pipeline to run for 1 hour 
> 2. Start the pipeline 
> 3. After it starts materializing new actions, change the end time (in my example - to 10 minutes after the pipeline 
> starts) 
> 4. Monitor the coordinator apps - they will continue to materialize new actions past the end time.
> 
> 
> This addresses bug OOZIE-591.
>     https://issues.apache.org/jira/browse/OOZIE-591
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/Command.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/XCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobChangeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobResumeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobSuspendXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobsXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleKillXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePauseXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePurgeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleRerunXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStartXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStatusUpdateXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleSubmitXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleUnpauseXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionCheckXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInfoXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionNotificationXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionReadyXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionStartXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionTimeOutXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdateXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordChangeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordJobXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordJobsXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordKillXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPauseXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPurgeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordRerunXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordResumeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSubmitXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSuspendXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordUnpauseXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/SLAEventsXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionCheckXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionEndXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionKillXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionStartXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/CompletedActionXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/DefinitionXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ExternalIdXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/JobXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/JobsXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/KillXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/NotificationXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/PurgeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ReRunXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ResumeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SignalXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SubmitMRXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SubmitPigXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SubmitXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SuspendXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WfEndXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WorkflowActionInfoXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/RecoveryService.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/XCallable.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/resources/oozie-default.xml 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionInputCheckXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionStartXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestCallableQueueService.java 1230165 
> 
> Diff: https://reviews.apache.org/r/3030/diff
> 
> 
> Testing
> -------
> 
> Regression for all the Commands in addition for specific testing for the Interrupt Driven Map
> 
> 
> Thanks,
> 
> Mohamed
> 
>


Re: Review Request: OOZIE-591: Oozie continues to materialize new actions after end date modification

Posted by Mohamed Battisha <mb...@yahoo.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3030/
-----------------------------------------------------------

(Updated 2012-01-13 01:18:34.385617)


Review request for oozie.


Changes
-------

fixing a copy and past error


Summary
-------

Oozie continues to materialize new actions after end date modification. The main issue is related how Oozie materialized the actions based on a FIFO priority queue. Changing the status of a bundle/coordinator job should take higher priority than executing this job. 

The main idea is to enable the queue to handle interruptions. Mainly, once you finished what you are working on currently, you should focus on executing this next action. 

The issue can be illustrated as follow: 

1. Configure a pipeline to run for 1 hour 
2. Start the pipeline 
3. After it starts materializing new actions, change the end time (in my example - to 10 minutes after the pipeline 
starts) 
4. Monitor the coordinator apps - they will continue to materialize new actions past the end time.


This addresses bug OOZIE-591.
    https://issues.apache.org/jira/browse/OOZIE-591


Diffs (updated)
-----

  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/Command.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/XCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobChangeXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobResumeXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobSuspendXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobsXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleKillXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePauseXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePurgeXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleRerunXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStartXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStatusUpdateXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleSubmitXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleUnpauseXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionCheckXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInfoXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionNotificationXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionReadyXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionStartXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionTimeOutXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdateXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordChangeXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordJobXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordJobsXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordKillXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPauseXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPurgeXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordRerunXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordResumeXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSubmitXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSuspendXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordUnpauseXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/SLAEventsXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionCheckXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionEndXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionKillXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionStartXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/CompletedActionXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/DefinitionXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ExternalIdXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/JobXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/JobsXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/KillXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/NotificationXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/PurgeXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ReRunXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ResumeXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SignalXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SubmitMRXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SubmitPigXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SubmitXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SuspendXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WfEndXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WorkflowActionInfoXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/RecoveryService.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/XCallable.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/resources/oozie-default.xml 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionInputCheckXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionStartXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestCallableQueueService.java 1230165 

Diff: https://reviews.apache.org/r/3030/diff


Testing
-------

Regression for all the Commands in addition for specific testing for the Interrupt Driven Map


Thanks,

Mohamed


Re: Review Request: OOZIE-591: Oozie continues to materialize new actions after end date modification

Posted by Mohamed Battisha <mb...@yahoo.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3030/
-----------------------------------------------------------

(Updated 2012-01-13 01:01:22.948842)


Review request for oozie.


Summary
-------

Oozie continues to materialize new actions after end date modification. The main issue is related how Oozie materialized the actions based on a FIFO priority queue. Changing the status of a bundle/coordinator job should take higher priority than executing this job. 

The main idea is to enable the queue to handle interruptions. Mainly, once you finished what you are working on currently, you should focus on executing this next action. 

The issue can be illustrated as follow: 

1. Configure a pipeline to run for 1 hour 
2. Start the pipeline 
3. After it starts materializing new actions, change the end time (in my example - to 10 minutes after the pipeline 
starts) 
4. Monitor the coordinator apps - they will continue to materialize new actions past the end time.


This addresses bug OOZIE-591.
    https://issues.apache.org/jira/browse/OOZIE-591


Diffs (updated)
-----

  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/Command.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/XCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobChangeXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobResumeXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobSuspendXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobsXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleKillXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePauseXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePurgeXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleRerunXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStartXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStatusUpdateXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleSubmitXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleUnpauseXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionCheckXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInfoXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionNotificationXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionReadyXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionStartXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionTimeOutXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdateXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordChangeXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordJobXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordJobsXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordKillXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPauseXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPurgeXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordRerunXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordResumeXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSubmitXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSuspendXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordUnpauseXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/SLAEventsXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionCheckXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionEndXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionKillXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionStartXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/CompletedActionXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/DefinitionXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ExternalIdXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/JobXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/JobsXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/KillXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/NotificationXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/PurgeXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ReRunXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ResumeXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SignalXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SubmitMRXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SubmitPigXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SubmitXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SuspendXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WfEndXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WorkflowActionInfoXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/RecoveryService.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/XCallable.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/resources/oozie-default.xml 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionInputCheckXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionStartXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestCallableQueueService.java 1230165 

Diff: https://reviews.apache.org/r/3030/diff


Testing
-------

Regression for all the Commands in addition for specific testing for the Interrupt Driven Map


Thanks,

Mohamed


Re: Review Request: OOZIE-591: Oozie continues to materialize new actions after end date modification

Posted by Mohamed Battisha <mb...@yahoo.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3030/
-----------------------------------------------------------

(Updated 2012-01-12 21:36:09.784944)


Review request for oozie.


Changes
-------

updated 


Summary
-------

Oozie continues to materialize new actions after end date modification. The main issue is related how Oozie materialized the actions based on a FIFO priority queue. Changing the status of a bundle/coordinator job should take higher priority than executing this job. 

The main idea is to enable the queue to handle interruptions. Mainly, once you finished what you are working on currently, you should focus on executing this next action. 

The issue can be illustrated as follow: 

1. Configure a pipeline to run for 1 hour 
2. Start the pipeline 
3. After it starts materializing new actions, change the end time (in my example - to 10 minutes after the pipeline 
starts) 
4. Monitor the coordinator apps - they will continue to materialize new actions past the end time.


This addresses bug OOZIE-591.
    https://issues.apache.org/jira/browse/OOZIE-591


Diffs (updated)
-----

  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/Command.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/XCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobChangeXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobResumeXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobSuspendXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobsXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleKillXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePauseXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePurgeXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleRerunXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStartXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStatusUpdateXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleSubmitXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleUnpauseXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionCheckXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInfoXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionNotificationXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionReadyXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionStartXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionTimeOutXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdateXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordChangeXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordJobXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordJobsXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordKillXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPauseXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPurgeXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordRerunXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordResumeXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSubmitXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSuspendXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordUnpauseXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/SLAEventsXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionCheckXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionEndXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionKillXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionStartXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/CompletedActionXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/DefinitionXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ExternalIdXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/JobXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/JobsXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/KillXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/NotificationXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/PurgeXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ReRunXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ResumeXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SignalXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SubmitMRXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SubmitPigXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SubmitXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SuspendXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WfEndXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WorkflowActionInfoXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/RecoveryService.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/XCallable.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/resources/oozie-default.xml 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionInputCheckXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionStartXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestCallableQueueService.java 1230165 

Diff: https://reviews.apache.org/r/3030/diff


Testing
-------

Regression for all the Commands in addition for specific testing for the Interrupt Driven Map


Thanks,

Mohamed


Re: Review Request: OOZIE-591: Oozie continues to materialize new actions after end date modification

Posted by Mohamed Battisha <mb...@yahoo.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3030/
-----------------------------------------------------------

(Updated 2012-01-12 03:04:27.489622)


Review request for oozie.


Changes
-------

updated based on reviewer feedback


Summary
-------

Oozie continues to materialize new actions after end date modification. The main issue is related how Oozie materialized the actions based on a FIFO priority queue. Changing the status of a bundle/coordinator job should take higher priority than executing this job. 

The main idea is to enable the queue to handle interruptions. Mainly, once you finished what you are working on currently, you should focus on executing this next action. 

The issue can be illustrated as follow: 

1. Configure a pipeline to run for 1 hour 
2. Start the pipeline 
3. After it starts materializing new actions, change the end time (in my example - to 10 minutes after the pipeline 
starts) 
4. Monitor the coordinator apps - they will continue to materialize new actions past the end time.


This addresses bug OOZIE-591.
    https://issues.apache.org/jira/browse/OOZIE-591


Diffs (updated)
-----

  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/Command.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/XCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobChangeXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobResumeXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobSuspendXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobsXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleKillXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePauseXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePurgeXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleRerunXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStartXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStatusUpdateXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleSubmitXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleUnpauseXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionCheckXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInfoXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionNotificationXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionReadyXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionStartXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionTimeOutXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdateXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordChangeXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordJobXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordJobsXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordKillXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPauseXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPurgeXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordRerunXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordResumeXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSubmitXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSuspendXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordUnpauseXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/SLAEventsXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionCheckXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionEndXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionKillXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionStartXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/CompletedActionXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/DefinitionXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ExternalIdXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/JobXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/JobsXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/KillXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/NotificationXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/PurgeXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ReRunXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ResumeXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SignalXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SubmitMRXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SubmitPigXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SubmitXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SuspendXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WfEndXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WorkflowActionInfoXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/RecoveryService.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/XCallable.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/resources/oozie-default.xml 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionInputCheckXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionStartXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestCallableQueueService.java 1230165 

Diff: https://reviews.apache.org/r/3030/diff


Testing
-------

Regression for all the Commands in addition for specific testing for the Interrupt Driven Map


Thanks,

Mohamed


Re: Review Request: OOZIE-591: Oozie continues to materialize new actions after end date modification

Posted by Mohamed Battisha <mb...@yahoo.com>.

> On 2012-01-11 21:09:44, Alejandro Abdelnur wrote:
> > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/XCommand.java, line 325
> > <https://reviews.apache.org/r/3030/diff/7/?file=67975#file67975line325>
> >
> >     what happens here if a new interrupt for the entity is being added while looping?

The list will be removed from the map before this loop
>> CallableQueueService @line 690
        if (lockKey != null) {
            return interruptCommandsMap.remove(lockKey);
        }

so if a new interrupt is added, a new list will be added to the map containing this new interrupt.


> On 2012-01-11 21:09:44, Alejandro Abdelnur wrote:
> > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java, line 731
> > <https://reviews.apache.org/r/3030/diff/7/?file=68031#file68031line731>
> >
> >     instead doing a get/ifNULL/put to the concurrent map use the putIfAbsent() method:
> >     
> >     ArrayList<XCallable<?>> list = map.putIfAbsent(callable.getEntity(), new ArrayList<XCallable<?>>());
> >     list.add(callable):
> >     
> >     this will avoid race conditions on creating the arraylist
> >     
> >     the other thing is that the ArrayList is no threadsafe, thus the addition has to be done in to wrap the list with Collections.synchronizedList(). this would not create contention across the system as the syncronization will be for a entity which is lock 'snychronized'

make sense, will consider it


- Mohamed


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3030/#review4316
-----------------------------------------------------------


On 2012-01-11 20:07:01, Mohamed Battisha wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/3030/
> -----------------------------------------------------------
> 
> (Updated 2012-01-11 20:07:01)
> 
> 
> Review request for oozie.
> 
> 
> Summary
> -------
> 
> Oozie continues to materialize new actions after end date modification. The main issue is related how Oozie materialized the actions based on a FIFO priority queue. Changing the status of a bundle/coordinator job should take higher priority than executing this job. 
> 
> The main idea is to enable the queue to handle interruptions. Mainly, once you finished what you are working on currently, you should focus on executing this next action. 
> 
> The issue can be illustrated as follow: 
> 
> 1. Configure a pipeline to run for 1 hour 
> 2. Start the pipeline 
> 3. After it starts materializing new actions, change the end time (in my example - to 10 minutes after the pipeline 
> starts) 
> 4. Monitor the coordinator apps - they will continue to materialize new actions past the end time.
> 
> 
> This addresses bug OOZIE-591.
>     https://issues.apache.org/jira/browse/OOZIE-591
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/Command.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/XCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobChangeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobResumeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobSuspendXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobsXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleKillXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePauseXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePurgeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleRerunXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStartXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStatusUpdateXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleSubmitXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleUnpauseXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionCheckXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInfoXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionNotificationXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionReadyXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionStartXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionTimeOutXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdateXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordChangeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordJobXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordJobsXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordKillXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPauseXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPurgeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordRerunXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordResumeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSubmitXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSuspendXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordUnpauseXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/SLAEventsXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionCheckXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionEndXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionKillXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionStartXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/CompletedActionXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/DefinitionXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ExternalIdXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/JobXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/JobsXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/KillXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/NotificationXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/PurgeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ReRunXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ResumeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SignalXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SubmitMRXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SubmitPigXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SubmitXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SuspendXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WfEndXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WorkflowActionInfoXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/RecoveryService.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/XCallable.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/resources/oozie-default.xml 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionInputCheckXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionStartXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestCallableQueueService.java 1230165 
> 
> Diff: https://reviews.apache.org/r/3030/diff
> 
> 
> Testing
> -------
> 
> Regression for all the Commands in addition for specific testing for the Interrupt Driven Map
> 
> 
> Thanks,
> 
> Mohamed
> 
>


Re: Review Request: OOZIE-591: Oozie continues to materialize new actions after end date modification

Posted by Alejandro Abdelnur <tu...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3030/#review4316
-----------------------------------------------------------


changes look good. a couple of follow ups.

thanks


http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/XCommand.java
<https://reviews.apache.org/r/3030/#comment9713>

    what happens here if a new interrupt for the entity is being added while looping?



http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java
<https://reviews.apache.org/r/3030/#comment9712>

    instead doing a get/ifNULL/put to the concurrent map use the putIfAbsent() method:
    
    ArrayList<XCallable<?>> list = map.putIfAbsent(callable.getEntity(), new ArrayList<XCallable<?>>());
    list.add(callable):
    
    this will avoid race conditions on creating the arraylist
    
    the other thing is that the ArrayList is no threadsafe, thus the addition has to be done in to wrap the list with Collections.synchronizedList(). this would not create contention across the system as the syncronization will be for a entity which is lock 'snychronized'


- Alejandro


On 2012-01-11 20:07:01, Mohamed Battisha wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/3030/
> -----------------------------------------------------------
> 
> (Updated 2012-01-11 20:07:01)
> 
> 
> Review request for oozie.
> 
> 
> Summary
> -------
> 
> Oozie continues to materialize new actions after end date modification. The main issue is related how Oozie materialized the actions based on a FIFO priority queue. Changing the status of a bundle/coordinator job should take higher priority than executing this job. 
> 
> The main idea is to enable the queue to handle interruptions. Mainly, once you finished what you are working on currently, you should focus on executing this next action. 
> 
> The issue can be illustrated as follow: 
> 
> 1. Configure a pipeline to run for 1 hour 
> 2. Start the pipeline 
> 3. After it starts materializing new actions, change the end time (in my example - to 10 minutes after the pipeline 
> starts) 
> 4. Monitor the coordinator apps - they will continue to materialize new actions past the end time.
> 
> 
> This addresses bug OOZIE-591.
>     https://issues.apache.org/jira/browse/OOZIE-591
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/Command.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/XCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobChangeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobResumeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobSuspendXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobsXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleKillXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePauseXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePurgeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleRerunXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStartXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStatusUpdateXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleSubmitXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleUnpauseXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionCheckXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInfoXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionNotificationXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionReadyXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionStartXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionTimeOutXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdateXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordChangeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordJobXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordJobsXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordKillXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPauseXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPurgeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordRerunXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordResumeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSubmitXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSuspendXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordUnpauseXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/SLAEventsXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionCheckXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionEndXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionKillXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionStartXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/CompletedActionXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/DefinitionXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ExternalIdXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/JobXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/JobsXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/KillXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/NotificationXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/PurgeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ReRunXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ResumeXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SignalXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SubmitMRXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SubmitPigXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SubmitXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SuspendXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WfEndXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WorkflowActionInfoXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/RecoveryService.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/XCallable.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/resources/oozie-default.xml 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionInputCheckXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionStartXCommand.java 1230165 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestCallableQueueService.java 1230165 
> 
> Diff: https://reviews.apache.org/r/3030/diff
> 
> 
> Testing
> -------
> 
> Regression for all the Commands in addition for specific testing for the Interrupt Driven Map
> 
> 
> Thanks,
> 
> Mohamed
> 
>


Re: Review Request: OOZIE-591: Oozie continues to materialize new actions after end date modification

Posted by Mohamed Battisha <mb...@yahoo.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3030/
-----------------------------------------------------------

(Updated 2012-01-11 20:07:01.503599)


Review request for oozie.


Changes
-------

Exposing the getEntityKey() and changing its signature in the XCommands


Summary
-------

Oozie continues to materialize new actions after end date modification. The main issue is related how Oozie materialized the actions based on a FIFO priority queue. Changing the status of a bundle/coordinator job should take higher priority than executing this job. 

The main idea is to enable the queue to handle interruptions. Mainly, once you finished what you are working on currently, you should focus on executing this next action. 

The issue can be illustrated as follow: 

1. Configure a pipeline to run for 1 hour 
2. Start the pipeline 
3. After it starts materializing new actions, change the end time (in my example - to 10 minutes after the pipeline 
starts) 
4. Monitor the coordinator apps - they will continue to materialize new actions past the end time.


This addresses bug OOZIE-591.
    https://issues.apache.org/jira/browse/OOZIE-591


Diffs (updated)
-----

  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/Command.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/XCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobChangeXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobResumeXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobSuspendXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobsXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleKillXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePauseXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePurgeXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleRerunXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStartXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStatusUpdateXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleSubmitXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleUnpauseXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionCheckXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInfoXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionNotificationXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionReadyXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionStartXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionTimeOutXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdateXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordChangeXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordJobXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordJobsXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordKillXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPauseXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPurgeXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordRerunXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordResumeXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSubmitXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSuspendXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordUnpauseXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/SLAEventsXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionCheckXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionEndXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionKillXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ActionStartXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/CompletedActionXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/DefinitionXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ExternalIdXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/JobXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/JobsXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/KillXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/NotificationXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/PurgeXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ReRunXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ResumeXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SignalXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SubmitMRXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SubmitPigXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SubmitXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/SuspendXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WfEndXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WorkflowActionInfoXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/RecoveryService.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/XCallable.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/resources/oozie-default.xml 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionInputCheckXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionStartXCommand.java 1230165 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestCallableQueueService.java 1230165 

Diff: https://reviews.apache.org/r/3030/diff


Testing
-------

Regression for all the Commands in addition for specific testing for the Interrupt Driven Map


Thanks,

Mohamed


Re: Review Request: OOZIE-591: Oozie continues to materialize new actions after end date modification

Posted by Mohamed Battisha <mb...@yahoo.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3030/
-----------------------------------------------------------

(Updated 2012-01-09 19:19:08.723644)


Review request for oozie.


Changes
-------

removing the null checking for the getEntityKey()


Summary
-------

Oozie continues to materialize new actions after end date modification. The main issue is related how Oozie materialized the actions based on a FIFO priority queue. Changing the status of a bundle/coordinator job should take higher priority than executing this job. 

The main idea is to enable the queue to handle interruptions. Mainly, once you finished what you are working on currently, you should focus on executing this next action. 

The issue can be illustrated as follow: 

1. Configure a pipeline to run for 1 hour 
2. Start the pipeline 
3. After it starts materializing new actions, change the end time (in my example - to 10 minutes after the pipeline 
starts) 
4. Monitor the coordinator apps - they will continue to materialize new actions past the end time.


This addresses bug OOZIE-591.
    https://issues.apache.org/jira/browse/OOZIE-591


Diffs (updated)
-----

  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/Command.java 1228133 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/XCommand.java 1228133 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java 1228133 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionReadyXCommand.java 1228133 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionStartXCommand.java 1228133 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java 1228133 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordRerunXCommand.java 1228133 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java 1228133 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/RecoveryService.java 1228133 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/XCallable.java 1228133 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/resources/oozie-default.xml 1228133 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestCommand.java 1228133 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionInputCheckXCommand.java 1228133 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionStartXCommand.java 1228133 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestCallableQueueService.java 1228133 

Diff: https://reviews.apache.org/r/3030/diff


Testing
-------

Regression for all the Commands in addition for specific testing for the Interrupt Driven Map


Thanks,

Mohamed


Re: Review Request: OOZIE-591: Oozie continues to materialize new actions after end date modification

Posted by Mohamed Battisha <mb...@yahoo.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3030/
-----------------------------------------------------------

(Updated 2012-01-06 13:45:50.382177)


Review request for oozie.


Changes
-------

Updated based on reviewers feedback 

Thanks!
Mohamed


Summary
-------

Oozie continues to materialize new actions after end date modification. The main issue is related how Oozie materialized the actions based on a FIFO priority queue. Changing the status of a bundle/coordinator job should take higher priority than executing this job. 

The main idea is to enable the queue to handle interruptions. Mainly, once you finished what you are working on currently, you should focus on executing this next action. 

The issue can be illustrated as follow: 

1. Configure a pipeline to run for 1 hour 
2. Start the pipeline 
3. After it starts materializing new actions, change the end time (in my example - to 10 minutes after the pipeline 
starts) 
4. Monitor the coordinator apps - they will continue to materialize new actions past the end time.


This addresses bug OOZIE-591.
    https://issues.apache.org/jira/browse/OOZIE-591


Diffs (updated)
-----

  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/Command.java 1228133 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/XCommand.java 1228133 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePauseXCommand.java 1228133 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStatusUpdateXCommand.java 1228133 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleUnpauseXCommand.java 1228133 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java 1228133 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionNotificationXCommand.java 1228133 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionReadyXCommand.java 1228133 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionStartXCommand.java 1228133 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionTimeOutXCommand.java 1228133 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdateXCommand.java 1228133 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java 1228133 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPauseXCommand.java 1228133 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordRerunXCommand.java 1228133 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordUnpauseXCommand.java 1228133 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WfEndXCommand.java 1228133 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java 1228133 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/RecoveryService.java 1228133 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/XCallable.java 1228133 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/resources/oozie-default.xml 1228133 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestCommand.java 1228133 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionInputCheckXCommand.java 1228133 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionStartXCommand.java 1228133 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestCallableQueueService.java 1228133 

Diff: https://reviews.apache.org/r/3030/diff


Testing
-------

Regression for all the Commands in addition for specific testing for the Interrupt Driven Map


Thanks,

Mohamed


Re: Review Request: OOZIE-591: Oozie continues to materialize new actions after end date modification

Posted by Mohamed Battisha <mb...@yahoo.com>.

> On 2012-01-05 22:02:09, Mohammad Islam wrote:
> > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/resources/oozie-default.xml, line 276
> > <https://reviews.apache.org/r/3030/diff/4/?file=66785#file66785line276>
> >
> >     Default 100 looks small. What about 500?
> >     Because one bundle kill could generate lot of kill commands from coord and WF level.

done


> On 2012-01-05 22:02:09, Mohammad Islam wrote:
> > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/resources/oozie-default.xml, line 1121
> > <https://reviews.apache.org/r/3030/diff/4/?file=66785#file66785line1121>
> >
> >     what is this for?
> >     I believe it was part of pig stats.

It is not part of my changes. It is already committed to the trunk before.  


> On 2012-01-05 22:02:09, Mohammad Islam wrote:
> > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/resources/oozie-default.xml, line 286
> > <https://reviews.apache.org/r/3030/diff/4/?file=66785#file66785line286>
> >
> >     does it cover bundle kill, coord kill, all syspends/resume, pause/unpause too

i did not add the full list of interrupts as it is configurable and can be changed  based on the deployment
that said, i will add these commands as well


> On 2012-01-05 22:02:09, Mohammad Islam wrote:
> > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPauseXCommand.java, line 48
> > <https://reviews.apache.org/r/3030/diff/4/?file=66775#file66775line48>
> >
> >     Although it is not part of this JIRA, I think the best approach to check the null in the constructor around line 41.

During my unit-test i faced several null pointer exception, while trying to access the entity key.
I preferred to have the check getEntityKey method as it is related to this JIRA. 
we may open another JARA to generalize it or do it in the constructor


> On 2012-01-05 22:02:09, Mohammad Islam wrote:
> > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionTimeOutXCommand.java, line 71
> > <https://reviews.apache.org/r/3030/diff/4/?file=66772#file66772line71>
> >
> >     null checking is done on constructor line 43.
> >     do we need it here too?
> >

nope, for this case it is not 
in some other changes, it may happen as it is not checked in the constructor 
during my unit-test i faced several null pointer exception. i just added them all over for consistency 
I can open another JIRA to address this issue in a more broad way


- Mohamed


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3030/#review4212
-----------------------------------------------------------


On 2012-01-05 20:12:58, Mohamed Battisha wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/3030/
> -----------------------------------------------------------
> 
> (Updated 2012-01-05 20:12:58)
> 
> 
> Review request for oozie.
> 
> 
> Summary
> -------
> 
> Oozie continues to materialize new actions after end date modification. The main issue is related how Oozie materialized the actions based on a FIFO priority queue. Changing the status of a bundle/coordinator job should take higher priority than executing this job. 
> 
> The main idea is to enable the queue to handle interruptions. Mainly, once you finished what you are working on currently, you should focus on executing this next action. 
> 
> The issue can be illustrated as follow: 
> 
> 1. Configure a pipeline to run for 1 hour 
> 2. Start the pipeline 
> 3. After it starts materializing new actions, change the end time (in my example - to 10 minutes after the pipeline 
> starts) 
> 4. Monitor the coordinator apps - they will continue to materialize new actions past the end time.
> 
> 
> This addresses bug OOZIE-591.
>     https://issues.apache.org/jira/browse/OOZIE-591
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/Command.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/XCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobResumeXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleRerunXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStartXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStatusUpdateXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleUnpauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionNotificationXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionReadyXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionStartXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionTimeOutXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdateXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordRerunXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordResumeXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSuspendXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordUnpauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ResumeXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WfEndXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/RecoveryService.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/XCallable.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/resources/oozie-default.xml 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionInputCheckXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionStartXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestCallableQueueService.java 1209829 
> 
> Diff: https://reviews.apache.org/r/3030/diff
> 
> 
> Testing
> -------
> 
> Regression for all the Commands in addition for specific testing for the Interrupt Driven Map
> 
> 
> Thanks,
> 
> Mohamed
> 
>


Re: Review Request: OOZIE-591: Oozie continues to materialize new actions after end date modification

Posted by Mohammad Islam <mi...@yahoo.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3030/#review4212
-----------------------------------------------------------



http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionTimeOutXCommand.java
<https://reviews.apache.org/r/3030/#comment9497>

    null checking is done on constructor line 43.
    do we need it here too?
    



http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPauseXCommand.java
<https://reviews.apache.org/r/3030/#comment9499>

    Although it is not part of this JIRA, I think the best approach to check the null in the constructor around line 41.



http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/resources/oozie-default.xml
<https://reviews.apache.org/r/3030/#comment9503>

    Default 100 looks small. What about 500?
    Because one bundle kill could generate lot of kill commands from coord and WF level.



http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/resources/oozie-default.xml
<https://reviews.apache.org/r/3030/#comment9504>

    does it cover bundle kill, coord kill, all syspends/resume, pause/unpause too



http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/resources/oozie-default.xml
<https://reviews.apache.org/r/3030/#comment9505>

    what is this for?
    I believe it was part of pig stats.


- Mohammad


On 2012-01-05 20:12:58, Mohamed Battisha wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/3030/
> -----------------------------------------------------------
> 
> (Updated 2012-01-05 20:12:58)
> 
> 
> Review request for oozie.
> 
> 
> Summary
> -------
> 
> Oozie continues to materialize new actions after end date modification. The main issue is related how Oozie materialized the actions based on a FIFO priority queue. Changing the status of a bundle/coordinator job should take higher priority than executing this job. 
> 
> The main idea is to enable the queue to handle interruptions. Mainly, once you finished what you are working on currently, you should focus on executing this next action. 
> 
> The issue can be illustrated as follow: 
> 
> 1. Configure a pipeline to run for 1 hour 
> 2. Start the pipeline 
> 3. After it starts materializing new actions, change the end time (in my example - to 10 minutes after the pipeline 
> starts) 
> 4. Monitor the coordinator apps - they will continue to materialize new actions past the end time.
> 
> 
> This addresses bug OOZIE-591.
>     https://issues.apache.org/jira/browse/OOZIE-591
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/Command.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/XCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobResumeXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleRerunXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStartXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStatusUpdateXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleUnpauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionNotificationXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionReadyXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionStartXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionTimeOutXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdateXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordRerunXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordResumeXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSuspendXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordUnpauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ResumeXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WfEndXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/RecoveryService.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/XCallable.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/resources/oozie-default.xml 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionInputCheckXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionStartXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestCallableQueueService.java 1209829 
> 
> Diff: https://reviews.apache.org/r/3030/diff
> 
> 
> Testing
> -------
> 
> Regression for all the Commands in addition for specific testing for the Interrupt Driven Map
> 
> 
> Thanks,
> 
> Mohamed
> 
>


Re: Review Request: OOZIE-591: Oozie continues to materialize new actions after end date modification

Posted by Mohamed Battisha <mb...@yahoo.com>.

> On 2012-01-05 22:21:17, Alejandro Abdelnur wrote:
> > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/resources/oozie-default.xml, line 1121
> > <https://reviews.apache.org/r/3030/diff/4/?file=66785#file66785line1121>
> >
> >     well, then it should not be part of this patch

It is not part of my changes. It is already committed to the trunk before.  


- Mohamed


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3030/#review4216
-----------------------------------------------------------


On 2012-01-05 20:12:58, Mohamed Battisha wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/3030/
> -----------------------------------------------------------
> 
> (Updated 2012-01-05 20:12:58)
> 
> 
> Review request for oozie.
> 
> 
> Summary
> -------
> 
> Oozie continues to materialize new actions after end date modification. The main issue is related how Oozie materialized the actions based on a FIFO priority queue. Changing the status of a bundle/coordinator job should take higher priority than executing this job. 
> 
> The main idea is to enable the queue to handle interruptions. Mainly, once you finished what you are working on currently, you should focus on executing this next action. 
> 
> The issue can be illustrated as follow: 
> 
> 1. Configure a pipeline to run for 1 hour 
> 2. Start the pipeline 
> 3. After it starts materializing new actions, change the end time (in my example - to 10 minutes after the pipeline 
> starts) 
> 4. Monitor the coordinator apps - they will continue to materialize new actions past the end time.
> 
> 
> This addresses bug OOZIE-591.
>     https://issues.apache.org/jira/browse/OOZIE-591
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/Command.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/XCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobResumeXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleRerunXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStartXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStatusUpdateXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleUnpauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionNotificationXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionReadyXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionStartXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionTimeOutXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdateXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordRerunXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordResumeXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSuspendXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordUnpauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ResumeXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WfEndXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/RecoveryService.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/XCallable.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/resources/oozie-default.xml 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionInputCheckXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionStartXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestCallableQueueService.java 1209829 
> 
> Diff: https://reviews.apache.org/r/3030/diff
> 
> 
> Testing
> -------
> 
> Regression for all the Commands in addition for specific testing for the Interrupt Driven Map
> 
> 
> Thanks,
> 
> Mohamed
> 
>


Re: Review Request: OOZIE-591: Oozie continues to materialize new actions after end date modification

Posted by Alejandro Abdelnur <tu...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3030/#review4216
-----------------------------------------------------------



http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/resources/oozie-default.xml
<https://reviews.apache.org/r/3030/#comment9507>

    well, then it should not be part of this patch


- Alejandro


On 2012-01-05 20:12:58, Mohamed Battisha wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/3030/
> -----------------------------------------------------------
> 
> (Updated 2012-01-05 20:12:58)
> 
> 
> Review request for oozie.
> 
> 
> Summary
> -------
> 
> Oozie continues to materialize new actions after end date modification. The main issue is related how Oozie materialized the actions based on a FIFO priority queue. Changing the status of a bundle/coordinator job should take higher priority than executing this job. 
> 
> The main idea is to enable the queue to handle interruptions. Mainly, once you finished what you are working on currently, you should focus on executing this next action. 
> 
> The issue can be illustrated as follow: 
> 
> 1. Configure a pipeline to run for 1 hour 
> 2. Start the pipeline 
> 3. After it starts materializing new actions, change the end time (in my example - to 10 minutes after the pipeline 
> starts) 
> 4. Monitor the coordinator apps - they will continue to materialize new actions past the end time.
> 
> 
> This addresses bug OOZIE-591.
>     https://issues.apache.org/jira/browse/OOZIE-591
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/Command.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/XCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobResumeXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleRerunXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStartXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStatusUpdateXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleUnpauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionNotificationXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionReadyXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionStartXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionTimeOutXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdateXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordRerunXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordResumeXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSuspendXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordUnpauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ResumeXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WfEndXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/RecoveryService.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/XCallable.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/resources/oozie-default.xml 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionInputCheckXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionStartXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestCallableQueueService.java 1209829 
> 
> Diff: https://reviews.apache.org/r/3030/diff
> 
> 
> Testing
> -------
> 
> Regression for all the Commands in addition for specific testing for the Interrupt Driven Map
> 
> 
> Thanks,
> 
> Mohamed
> 
>


Re: Review Request: OOZIE-591: Oozie continues to materialize new actions after end date modification

Posted by Mohamed Battisha <mb...@yahoo.com>.

> On 2012-01-05 21:04:45, Alejandro Abdelnur wrote:
> > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/XCommand.java, line 245
> > <https://reviews.apache.org/r/3030/diff/4/?file=66761#file66761line245>
> >
> >     adding to the comment of 3 different keys
> >     
> >     the fact that the lock is on a different key that the interruption it is an issue as the outer lock may be for a different key that the command being processed in the interruption

nope, the interrupt is using  the lock key ... only two keys (key and entityKey [lockKey])


> On 2012-01-05 21:04:45, Alejandro Abdelnur wrote:
> > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/XCommand.java, line 483
> > <https://reviews.apache.org/r/3030/diff/4/?file=66761#file66761line483>
> >
> >     This adds a third key to the XCommand
> >     
> >     getKey()
> >     getLockKey()
> >     getEntityKey()
> >     
> >     the acquireLock is done with the getEntityKey(), the getLock() is used for interruptions exclusively. 
> >     
> >     I see 2 issues here.
> >     
> >     1* the naming is confusing
> >     2* Do we really need 3 different keys?
> >

There are only two keys [key, entity key] in the XCommand
getLockKey() is mainly returning the getEntityKey()

The main goal is to make the entity key exposed in the XCallable interface and implement it in XCommand [this is why we introduce the getLockKey() method in the XCallable interface]
I understand the naming may be confusing, I will think about a different approach to eliminate this confusion


> On 2012-01-05 21:04:45, Alejandro Abdelnur wrote:
> > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/XCommand.java, line 492
> > <https://reviews.apache.org/r/3030/diff/4/?file=66761#file66761line492>
> >
> >     no need to use 'this.'

Yep, I understand but I just get used to put this before any private member. it make it easy for reading the code. it is a recommended standard for a lot of OO languages


> On 2012-01-05 21:04:45, Alejandro Abdelnur wrote:
> > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobResumeXCommand.java, line 137
> > <https://reviews.apache.org/r/3030/diff/4/?file=66762#file66762line137>
> >
> >     false change! (there are several of these through the patch)

will revert this file .. no need to have it for only this change


> On 2012-01-05 21:04:45, Alejandro Abdelnur wrote:
> > http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePauseXCommand.java, line 45
> > <https://reviews.apache.org/r/3030/diff/4/?file=66763#file66763line45>
> >
> >     is it posible/valid no to have a bundleJOb here? 
> >     
> >     same Q for similar changes in other commands

nope, for this case it is not 
in some other changes, during my unit-test i faced several null pointer exceptions. i just added them all over for consistency 
I can open another JIRA to address this issue in a more broad way


- Mohamed


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3030/#review4211
-----------------------------------------------------------


On 2012-01-05 20:12:58, Mohamed Battisha wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/3030/
> -----------------------------------------------------------
> 
> (Updated 2012-01-05 20:12:58)
> 
> 
> Review request for oozie.
> 
> 
> Summary
> -------
> 
> Oozie continues to materialize new actions after end date modification. The main issue is related how Oozie materialized the actions based on a FIFO priority queue. Changing the status of a bundle/coordinator job should take higher priority than executing this job. 
> 
> The main idea is to enable the queue to handle interruptions. Mainly, once you finished what you are working on currently, you should focus on executing this next action. 
> 
> The issue can be illustrated as follow: 
> 
> 1. Configure a pipeline to run for 1 hour 
> 2. Start the pipeline 
> 3. After it starts materializing new actions, change the end time (in my example - to 10 minutes after the pipeline 
> starts) 
> 4. Monitor the coordinator apps - they will continue to materialize new actions past the end time.
> 
> 
> This addresses bug OOZIE-591.
>     https://issues.apache.org/jira/browse/OOZIE-591
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/Command.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/XCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobResumeXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleRerunXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStartXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStatusUpdateXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleUnpauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionNotificationXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionReadyXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionStartXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionTimeOutXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdateXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordRerunXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordResumeXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSuspendXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordUnpauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ResumeXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WfEndXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/RecoveryService.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/XCallable.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/resources/oozie-default.xml 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionInputCheckXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionStartXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestCallableQueueService.java 1209829 
> 
> Diff: https://reviews.apache.org/r/3030/diff
> 
> 
> Testing
> -------
> 
> Regression for all the Commands in addition for specific testing for the Interrupt Driven Map
> 
> 
> Thanks,
> 
> Mohamed
> 
>


Re: Review Request: OOZIE-591: Oozie continues to materialize new actions after end date modification

Posted by Alejandro Abdelnur <tu...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3030/#review4211
-----------------------------------------------------------



http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/XCommand.java
<https://reviews.apache.org/r/3030/#comment9496>

    adding to the comment of 3 different keys
    
    the fact that the lock is on a different key that the interruption it is an issue as the outer lock may be for a different key that the command being processed in the interruption



http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/XCommand.java
<https://reviews.apache.org/r/3030/#comment9491>

    This adds a third key to the XCommand
    
    getKey()
    getLockKey()
    getEntityKey()
    
    the acquireLock is done with the getEntityKey(), the getLock() is used for interruptions exclusively. 
    
    I see 2 issues here.
    
    1* the naming is confusing
    2* Do we really need 3 different keys?
    



http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/XCommand.java
<https://reviews.apache.org/r/3030/#comment9492>

    no need to use 'this.'



http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/XCommand.java
<https://reviews.apache.org/r/3030/#comment9493>

    no need to use 'this.'



http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobResumeXCommand.java
<https://reviews.apache.org/r/3030/#comment9494>

    false change! (there are several of these through the patch)



http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePauseXCommand.java
<https://reviews.apache.org/r/3030/#comment9495>

    is it posible/valid no to have a bundleJOb here? 
    
    same Q for similar changes in other commands


- Alejandro


On 2012-01-05 20:12:58, Mohamed Battisha wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/3030/
> -----------------------------------------------------------
> 
> (Updated 2012-01-05 20:12:58)
> 
> 
> Review request for oozie.
> 
> 
> Summary
> -------
> 
> Oozie continues to materialize new actions after end date modification. The main issue is related how Oozie materialized the actions based on a FIFO priority queue. Changing the status of a bundle/coordinator job should take higher priority than executing this job. 
> 
> The main idea is to enable the queue to handle interruptions. Mainly, once you finished what you are working on currently, you should focus on executing this next action. 
> 
> The issue can be illustrated as follow: 
> 
> 1. Configure a pipeline to run for 1 hour 
> 2. Start the pipeline 
> 3. After it starts materializing new actions, change the end time (in my example - to 10 minutes after the pipeline 
> starts) 
> 4. Monitor the coordinator apps - they will continue to materialize new actions past the end time.
> 
> 
> This addresses bug OOZIE-591.
>     https://issues.apache.org/jira/browse/OOZIE-591
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/Command.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/XCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobResumeXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleRerunXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStartXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStatusUpdateXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleUnpauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionNotificationXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionReadyXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionStartXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionTimeOutXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdateXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordRerunXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordResumeXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSuspendXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordUnpauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ResumeXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WfEndXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/RecoveryService.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/XCallable.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/resources/oozie-default.xml 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionInputCheckXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionStartXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestCallableQueueService.java 1209829 
> 
> Diff: https://reviews.apache.org/r/3030/diff
> 
> 
> Testing
> -------
> 
> Regression for all the Commands in addition for specific testing for the Interrupt Driven Map
> 
> 
> Thanks,
> 
> Mohamed
> 
>


Re: Review Request: OOZIE-591: Oozie continues to materialize new actions after end date modification

Posted by Mohamed Battisha <mb...@yahoo.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3030/
-----------------------------------------------------------

(Updated 2012-01-05 20:12:58.753704)


Review request for oozie.


Changes
-------

Updated it based on the discussion with Mohammad and Alejandro 
Now the execution of the interrupt commands will take place in the XCommand instead of doing in the Pollable Queue.

Thanks!
Mohamed


Summary
-------

Oozie continues to materialize new actions after end date modification. The main issue is related how Oozie materialized the actions based on a FIFO priority queue. Changing the status of a bundle/coordinator job should take higher priority than executing this job. 

The main idea is to enable the queue to handle interruptions. Mainly, once you finished what you are working on currently, you should focus on executing this next action. 

The issue can be illustrated as follow: 

1. Configure a pipeline to run for 1 hour 
2. Start the pipeline 
3. After it starts materializing new actions, change the end time (in my example - to 10 minutes after the pipeline 
starts) 
4. Monitor the coordinator apps - they will continue to materialize new actions past the end time.


This addresses bug OOZIE-591.
    https://issues.apache.org/jira/browse/OOZIE-591


Diffs (updated)
-----

  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/Command.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/XCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobResumeXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePauseXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleRerunXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStartXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStatusUpdateXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleUnpauseXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionNotificationXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionReadyXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionStartXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionTimeOutXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdateXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPauseXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordRerunXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordResumeXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSuspendXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordUnpauseXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ResumeXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WfEndXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/RecoveryService.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/XCallable.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/resources/oozie-default.xml 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionInputCheckXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionStartXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestCallableQueueService.java 1209829 

Diff: https://reviews.apache.org/r/3030/diff


Testing
-------

Regression for all the Commands in addition for specific testing for the Interrupt Driven Map


Thanks,

Mohamed


Re: Review Request: OOZIE-591: Oozie continues to materialize new actions after end date modification

Posted by Mohamed Battisha <mb...@yahoo.com>.

> On 2011-12-09 21:59:30, Alejandro Abdelnur wrote:
> > I'm not understanding how this patch handles interruptions by just adding a new method to command.
> > 
> > From the description of the problem, my logical solution would be:
> > 
> > * kill/suspend/modify commands would be only commands with highest priority (either by bumping everybody else one level or create a higher priority) (logically this is the same as interruptions but using the current priority logic which we know it works fine)
> > * all commands should check in their precondition that the job is in the expected state (job status and that the command in the case of coordinator jobs is for an action before the end time. The later is the case when the endtime has been shorten) 
> > 
> >

I attached the design document to the Jira [I added it to the Jira couple of weeks ago], giving more clarity about the problem and the proposed solution 

The interrupt is mainly handled at the interrupt map, which will hold all the interrupt driven commands and give them higher priority over the other commands that belong to the same lock key. it has been done at similar fashion of the priority queue but taking care of which job "lock key" this command belongs 
The method is added to get the lock key of each command which is used by the interrupt driven map.


The idea you proposed will work, but it may cause starvation for the other jobs as you are not going to promote any commands to the higher priority queue and you are not executing it based on the jobs. 
so few jobs that have some interrupt commands "kill, suspend,modify" may cause the other jobs to starve while waiting to execute their commands in the higher priority queue.  


- Mohamed


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3030/#review3801
-----------------------------------------------------------


On 2011-12-09 18:58:56, Mohamed Battisha wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/3030/
> -----------------------------------------------------------
> 
> (Updated 2011-12-09 18:58:56)
> 
> 
> Review request for oozie.
> 
> 
> Summary
> -------
> 
> Oozie continues to materialize new actions after end date modification. The main issue is related how Oozie materialized the actions based on a FIFO priority queue. Changing the status of a bundle/coordinator job should take higher priority than executing this job. 
> 
> The main idea is to enable the queue to handle interruptions. Mainly, once you finished what you are working on currently, you should focus on executing this next action. 
> 
> The issue can be illustrated as follow: 
> 
> 1. Configure a pipeline to run for 1 hour 
> 2. Start the pipeline 
> 3. After it starts materializing new actions, change the end time (in my example - to 10 minutes after the pipeline 
> starts) 
> 4. Monitor the coordinator apps - they will continue to materialize new actions past the end time.
> 
> 
> This addresses bug OOZIE-591.
>     https://issues.apache.org/jira/browse/OOZIE-591
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/Command.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/XCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobResumeXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleRerunXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStartXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStatusUpdateXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleUnpauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionNotificationXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionReadyXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionStartXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionTimeOutXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdateXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordRerunXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordResumeXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSuspendXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordUnpauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ResumeXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WfEndXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/RecoveryService.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/PollablePriorityDelayQueue.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/XCallable.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/resources/oozie-default.xml 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionInputCheckXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionStartXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestCallableQueueService.java 1209829 
> 
> Diff: https://reviews.apache.org/r/3030/diff
> 
> 
> Testing
> -------
> 
> Regression for all the Commands in addition for specific testing for the Interrupt Driven Map
> 
> 
> Thanks,
> 
> Mohamed
> 
>


Re: Review Request: OOZIE-591: Oozie continues to materialize new actions after end date modification

Posted by Alejandro Abdelnur <tu...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3030/#review3801
-----------------------------------------------------------


I'm not understanding how this patch handles interruptions by just adding a new method to command.

>From the description of the problem, my logical solution would be:

* kill/suspend/modify commands would be only commands with highest priority (either by bumping everybody else one level or create a higher priority) (logically this is the same as interruptions but using the current priority logic which we know it works fine)
* all commands should check in their precondition that the job is in the expected state (job status and that the command in the case of coordinator jobs is for an action before the end time. The later is the case when the endtime has been shorten) 



- Alejandro


On 2011-12-09 18:58:56, Mohamed Battisha wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/3030/
> -----------------------------------------------------------
> 
> (Updated 2011-12-09 18:58:56)
> 
> 
> Review request for oozie.
> 
> 
> Summary
> -------
> 
> Oozie continues to materialize new actions after end date modification. The main issue is related how Oozie materialized the actions based on a FIFO priority queue. Changing the status of a bundle/coordinator job should take higher priority than executing this job. 
> 
> The main idea is to enable the queue to handle interruptions. Mainly, once you finished what you are working on currently, you should focus on executing this next action. 
> 
> The issue can be illustrated as follow: 
> 
> 1. Configure a pipeline to run for 1 hour 
> 2. Start the pipeline 
> 3. After it starts materializing new actions, change the end time (in my example - to 10 minutes after the pipeline 
> starts) 
> 4. Monitor the coordinator apps - they will continue to materialize new actions past the end time.
> 
> 
> This addresses bug OOZIE-591.
>     https://issues.apache.org/jira/browse/OOZIE-591
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/Command.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/XCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobResumeXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleRerunXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStartXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStatusUpdateXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleUnpauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionNotificationXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionReadyXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionStartXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionTimeOutXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdateXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordRerunXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordResumeXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSuspendXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordUnpauseXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ResumeXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WfEndXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/RecoveryService.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/PollablePriorityDelayQueue.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/XCallable.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/resources/oozie-default.xml 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionInputCheckXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionStartXCommand.java 1209829 
>   http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestCallableQueueService.java 1209829 
> 
> Diff: https://reviews.apache.org/r/3030/diff
> 
> 
> Testing
> -------
> 
> Regression for all the Commands in addition for specific testing for the Interrupt Driven Map
> 
> 
> Thanks,
> 
> Mohamed
> 
>


Re: Review Request: OOZIE-591: Oozie continues to materialize new actions after end date modification

Posted by Mohamed Battisha <mb...@yahoo.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3030/
-----------------------------------------------------------

(Updated 2011-12-09 18:58:56.767449)


Review request for oozie.


Changes
-------

Updated to reflect the reviewers comments 


Summary
-------

Oozie continues to materialize new actions after end date modification. The main issue is related how Oozie materialized the actions based on a FIFO priority queue. Changing the status of a bundle/coordinator job should take higher priority than executing this job. 

The main idea is to enable the queue to handle interruptions. Mainly, once you finished what you are working on currently, you should focus on executing this next action. 

The issue can be illustrated as follow: 

1. Configure a pipeline to run for 1 hour 
2. Start the pipeline 
3. After it starts materializing new actions, change the end time (in my example - to 10 minutes after the pipeline 
starts) 
4. Monitor the coordinator apps - they will continue to materialize new actions past the end time.


This addresses bug OOZIE-591.
    https://issues.apache.org/jira/browse/OOZIE-591


Diffs (updated)
-----

  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/Command.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/XCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleJobResumeXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundlePauseXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleRerunXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStartXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleStatusUpdateXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/bundle/BundleUnpauseXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionNotificationXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionReadyXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionStartXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionTimeOutXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdateXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordPauseXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordRerunXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordResumeXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSuspendXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordUnpauseXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/ResumeXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/command/wf/WfEndXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/CallableQueueService.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/service/RecoveryService.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/PollablePriorityDelayQueue.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/java/org/apache/oozie/util/XCallable.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/main/resources/oozie-default.xml 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/TestCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionInputCheckXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordActionStartXCommand.java 1209829 
  http://svn.apache.org/repos/asf/incubator/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestCallableQueueService.java 1209829 

Diff: https://reviews.apache.org/r/3030/diff


Testing
-------

Regression for all the Commands in addition for specific testing for the Interrupt Driven Map


Thanks,

Mohamed