You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by "Ross, Richard (Truven Health)" <ri...@truvenhealth.com> on 2013/11/04 19:22:09 UTC
Stopping Oozie Coordinator from running in the past
Hello:
I have been playing around with Oozie Coordinator and noticed that when a coordinator definition is submitted, if the start value is in the past, it will run the workflow based on this past start and frequency values to "catch up" to the current time. Is there a way to tell Oozie to not run any workflows from the past?
For example, if the actual time when the Coordinator definition is submitted is 2013-11-04T12:30Z and the coordinator.xml is:
<coordinator-app name="test-coord"
frequency="${coord:hours(1)}"
start="2013-11-04T00:00Z"
end="2013-11-04T24:00Z"
timezone="UTC" xmlns="uri:oozie:coordinator:0.1">
<action>
<workflow>
.
.
.
</workflow>
</action>
</coordinator-app>
then workflow will run from 2013-11-04T00:00Z to 2013-11-04T12:00Z until it catches up (i.e., 12 or 13 times. Not sure as this is an example) to the current time. It will then follow the schedule. I would like it to just execute the next time the schedule is satisfied (i.e., is 2013-11-04T13:00Z)
Thanks,
Richard.
Re: Stopping Oozie Coordinator from running in the past
Posted by Mohammad Islam <mi...@yahoo.com>.
This is a good proposal. The property could be defined in the control section of the specific coordinator.
Regards,
Mohammad
On Tuesday, November 5, 2013 6:09 AM, "Ross, Richard (Truven Health)" <ri...@truvenhealth.com> wrote:
Thanks, Mohammad. That's what I thought, but I just wanted to make sure I wasn't missing something.
In terms of the behavior, I would think that the "trigger" times when in NO_PAST_EXECUTION mode (bad name maybe, but you get the idea) would be a subset of those that would occur in the current (DEFAULT_EXECUTION) mode, irrespective of the submit time. So, continuing with the example that I included in my original email with submit time of 2013-11-04T12:30Z, the fire times would be:
DEFAULT_EXECUTION NO_PAST_EXECUTION
----------------- -----------------
2013-11-04T00:00Z
2013-11-04T01:00Z
2013-11-04T02:00Z
2013-11-04T03:00Z
2013-11-04T04:00Z
2013-11-04T05:00Z
2013-11-04T06:00Z
2013-11-04T07:00Z
2013-11-04T08:00Z
2013-11-04T09:00Z
2013-11-04T10:00Z
2013-11-04T11:00Z
2013-11-04T12:00Z
2013-11-04T13:00Z 2013-11-04T13:00Z
2013-11-04T14:00Z 2013-11-04T14:00Z
2013-11-04T15:00Z 2013-11-04T15:00Z
2013-11-04T16:00Z 2013-11-04T16:00Z
2013-11-04T17:00Z 2013-11-04T17:00Z
2013-11-04T18:00Z 2013-11-04T18:00Z
2013-11-04T19:00Z 2013-11-04T19:00Z
2013-11-04T20:00Z 2013-11-04T20:00Z
2013-11-04T21:00Z 2013-11-04T21:00Z
2013-11-04T22:00Z 2013-11-04T22:00Z
2013-11-04T23:00Z 2013-11-04T23:00Z
2013-11-04T24:00Z 2013-11-04T24:00Z
I have not run this example in the current Coordinator, so I'm not sure about the first and last times (whether they fire or not), but I think you get the idea.
Thanks again,
Richard.
-----Original Message-----
From: Mohammad Islam [mailto:mislam77@yahoo.com]
Sent: Monday, November 04, 2013 3:34 PM
To: user@oozie.apache.org
Subject: Re: Stopping Oozie Coordinator from running in the past
Hi Ross,
Currently there is no way of doing that.
The same discussion came few times before. I think Oozie should provide an easy way to start coordinator from the current time. However, when we talked previously, it was not easy to determine granularity of current time. For example, if the current time is :"2013-11-04 10:03Z" and the frequency is 5 minutes. What should be the action nominal time. Will it be 10:03, 10:08 or 10:13 etc. OR it will 10:00, 10:05, 10:10. The reason is : based on starttime the value of current() and latest() are calculated.
Having said that, we should start this discussion and come to an agreement. Any volunteer?
Regards,
Mohammad
On Monday, November 4, 2013 11:42 AM, "Ross, Richard (Truven Health)" <ri...@truvenhealth.com> wrote:
Hey Mona:
Thanks for your reply, and I agree that this should be parameterized (this was just an example). But I still have to make sure that the value of the parameters is correct and not in the past. In my mind I have just pushed the problem to another location. What I want to know is if there is a setting for the Coordinator that will make it so that I don't have to worry about this.
Thanks,
Richard.
-----Original Message-----
From: Mona Chitnis [mailto:chitnis@yahoo-inc.com]
Sent: Monday, November 04, 2013 1:47 PM
To: user@oozie.apache.org
Subject: Re: Stopping Oozie Coordinator from running in the past
Hi Richard,
Why not parameterize the coordinator "start" time? This start time should be when you wish the first instance to run.
E.g. <coordinator-app name="test-coord"
frequency="${coord:hours(1)}"
start=${my_start_time}
end="2013-11-04T24:00Z">
--
Mona
On 11/4/13 10:24 AM, "Ross, Richard (Truven Health)"
<ri...@truvenhealth.com> wrote:
>Forgot the Oozie version info: Oozie 3.3.2.
>
>Thanks,
>Richard.
>
>-----Original Message-----
>From: Ross, Richard (Truven Health)
>[mailto:richard.ross@truvenhealth.com]
>Sent: Monday, November 04, 2013 1:22 PM
>To: user@oozie.apache.org
>Subject: Stopping Oozie Coordinator from running in the past
>
>Hello:
>
>I have been playing around with Oozie Coordinator and noticed that when
>a coordinator definition is submitted, if the start value is in the
>past, it will run the workflow based on this past start and frequency
>values to "catch up" to the current time. Is there a way to tell Oozie
>to not run any workflows from the past?
>
>For example, if the actual time when the Coordinator definition is
>submitted is 2013-11-04T12:30Z and the coordinator.xml is:
>
><coordinator-app name="test-coord"
> frequency="${coord:hours(1)}"
> start="2013-11-04T00:00Z"
> end="2013-11-04T24:00Z"
> timezone="UTC" xmlns="uri:oozie:coordinator:0.1">
>
> <action>
> <workflow>
> .
> .
> .
> </workflow>
> </action>
></coordinator-app>
>
>then workflow will run from 2013-11-04T00:00Z to 2013-11-04T12:00Z
>until it catches up (i.e., 12 or 13 times. Not sure as this is an
>example) to the current time. It will then follow the schedule. I would
>like it to just execute the next time the schedule is satisfied (i.e.,
>is
>2013-11-04T13:00Z)
>
>Thanks,
>Richard.
RE: Stopping Oozie Coordinator from running in the past
Posted by "Ross, Richard (Truven Health)" <ri...@truvenhealth.com>.
Thanks, Mohammad. That's what I thought, but I just wanted to make sure I wasn't missing something.
In terms of the behavior, I would think that the "trigger" times when in NO_PAST_EXECUTION mode (bad name maybe, but you get the idea) would be a subset of those that would occur in the current (DEFAULT_EXECUTION) mode, irrespective of the submit time. So, continuing with the example that I included in my original email with submit time of 2013-11-04T12:30Z, the fire times would be:
DEFAULT_EXECUTION NO_PAST_EXECUTION
----------------- -----------------
2013-11-04T00:00Z
2013-11-04T01:00Z
2013-11-04T02:00Z
2013-11-04T03:00Z
2013-11-04T04:00Z
2013-11-04T05:00Z
2013-11-04T06:00Z
2013-11-04T07:00Z
2013-11-04T08:00Z
2013-11-04T09:00Z
2013-11-04T10:00Z
2013-11-04T11:00Z
2013-11-04T12:00Z
2013-11-04T13:00Z 2013-11-04T13:00Z
2013-11-04T14:00Z 2013-11-04T14:00Z
2013-11-04T15:00Z 2013-11-04T15:00Z
2013-11-04T16:00Z 2013-11-04T16:00Z
2013-11-04T17:00Z 2013-11-04T17:00Z
2013-11-04T18:00Z 2013-11-04T18:00Z
2013-11-04T19:00Z 2013-11-04T19:00Z
2013-11-04T20:00Z 2013-11-04T20:00Z
2013-11-04T21:00Z 2013-11-04T21:00Z
2013-11-04T22:00Z 2013-11-04T22:00Z
2013-11-04T23:00Z 2013-11-04T23:00Z
2013-11-04T24:00Z 2013-11-04T24:00Z
I have not run this example in the current Coordinator, so I'm not sure about the first and last times (whether they fire or not), but I think you get the idea.
Thanks again,
Richard.
-----Original Message-----
From: Mohammad Islam [mailto:mislam77@yahoo.com]
Sent: Monday, November 04, 2013 3:34 PM
To: user@oozie.apache.org
Subject: Re: Stopping Oozie Coordinator from running in the past
Hi Ross,
Currently there is no way of doing that.
The same discussion came few times before. I think Oozie should provide an easy way to start coordinator from the current time. However, when we talked previously, it was not easy to determine granularity of current time. For example, if the current time is :"2013-11-04 10:03Z" and the frequency is 5 minutes. What should be the action nominal time. Will it be 10:03, 10:08 or 10:13 etc. OR it will 10:00, 10:05, 10:10. The reason is : based on starttime the value of current() and latest() are calculated.
Having said that, we should start this discussion and come to an agreement. Any volunteer?
Regards,
Mohammad
On Monday, November 4, 2013 11:42 AM, "Ross, Richard (Truven Health)" <ri...@truvenhealth.com> wrote:
Hey Mona:
Thanks for your reply, and I agree that this should be parameterized (this was just an example). But I still have to make sure that the value of the parameters is correct and not in the past. In my mind I have just pushed the problem to another location. What I want to know is if there is a setting for the Coordinator that will make it so that I don't have to worry about this.
Thanks,
Richard.
-----Original Message-----
From: Mona Chitnis [mailto:chitnis@yahoo-inc.com]
Sent: Monday, November 04, 2013 1:47 PM
To: user@oozie.apache.org
Subject: Re: Stopping Oozie Coordinator from running in the past
Hi Richard,
Why not parameterize the coordinator "start" time? This start time should be when you wish the first instance to run.
E.g. <coordinator-app name="test-coord"
frequency="${coord:hours(1)}"
start=${my_start_time}
end="2013-11-04T24:00Z">
--
Mona
On 11/4/13 10:24 AM, "Ross, Richard (Truven Health)"
<ri...@truvenhealth.com> wrote:
>Forgot the Oozie version info: Oozie 3.3.2.
>
>Thanks,
>Richard.
>
>-----Original Message-----
>From: Ross, Richard (Truven Health)
>[mailto:richard.ross@truvenhealth.com]
>Sent: Monday, November 04, 2013 1:22 PM
>To: user@oozie.apache.org
>Subject: Stopping Oozie Coordinator from running in the past
>
>Hello:
>
>I have been playing around with Oozie Coordinator and noticed that when
>a coordinator definition is submitted, if the start value is in the
>past, it will run the workflow based on this past start and frequency
>values to "catch up" to the current time. Is there a way to tell Oozie
>to not run any workflows from the past?
>
>For example, if the actual time when the Coordinator definition is
>submitted is 2013-11-04T12:30Z and the coordinator.xml is:
>
><coordinator-app name="test-coord"
> frequency="${coord:hours(1)}"
> start="2013-11-04T00:00Z"
> end="2013-11-04T24:00Z"
> timezone="UTC" xmlns="uri:oozie:coordinator:0.1">
>
> <action>
> <workflow>
> .
> .
> .
> </workflow>
> </action>
></coordinator-app>
>
>then workflow will run from 2013-11-04T00:00Z to 2013-11-04T12:00Z
>until it catches up (i.e., 12 or 13 times. Not sure as this is an
>example) to the current time. It will then follow the schedule. I would
>like it to just execute the next time the schedule is satisfied (i.e.,
>is
>2013-11-04T13:00Z)
>
>Thanks,
>Richard.
Re: Stopping Oozie Coordinator from running in the past
Posted by Mohammad Islam <mi...@yahoo.com>.
Hi Ross,
Currently there is no way of doing that.
The same discussion came few times before. I think Oozie should provide an easy way to start coordinator from the current time.
However, when we talked previously, it was not easy to determine granularity of current time. For example, if the current time is :"2013-11-04 10:03Z" and the frequency is 5 minutes. What should be the action nominal time. Will it be 10:03, 10:08 or 10:13 etc. OR it will 10:00, 10:05, 10:10.
The reason is : based on starttime the value of current() and latest() are calculated.
Having said that, we should start this discussion and come to an agreement. Any volunteer?
Regards,
Mohammad
On Monday, November 4, 2013 11:42 AM, "Ross, Richard (Truven Health)" <ri...@truvenhealth.com> wrote:
Hey Mona:
Thanks for your reply, and I agree that this should be parameterized (this was just an example). But I still have to make sure that the value of the parameters is correct and not in the past. In my mind I have just pushed the problem to another location. What I want to know is if there is a setting for the Coordinator that will make it so that I don't have to worry about this.
Thanks,
Richard.
-----Original Message-----
From: Mona Chitnis [mailto:chitnis@yahoo-inc.com]
Sent: Monday, November 04, 2013 1:47 PM
To: user@oozie.apache.org
Subject: Re: Stopping Oozie Coordinator from running in the past
Hi Richard,
Why not parameterize the coordinator "start" time? This start time should be when you wish the first instance to run.
E.g. <coordinator-app name="test-coord"
frequency="${coord:hours(1)}"
start=${my_start_time}
end="2013-11-04T24:00Z">
--
Mona
On 11/4/13 10:24 AM, "Ross, Richard (Truven Health)"
<ri...@truvenhealth.com> wrote:
>Forgot the Oozie version info: Oozie 3.3.2.
>
>Thanks,
>Richard.
>
>-----Original Message-----
>From: Ross, Richard (Truven Health)
>[mailto:richard.ross@truvenhealth.com]
>Sent: Monday, November 04, 2013 1:22 PM
>To: user@oozie.apache.org
>Subject: Stopping Oozie Coordinator from running in the past
>
>Hello:
>
>I have been playing around with Oozie Coordinator and noticed that when
>a coordinator definition is submitted, if the start value is in the
>past, it will run the workflow based on this past start and frequency
>values to "catch up" to the current time. Is there a way to tell Oozie
>to not run any workflows from the past?
>
>For example, if the actual time when the Coordinator definition is
>submitted is 2013-11-04T12:30Z and the coordinator.xml is:
>
><coordinator-app name="test-coord"
> frequency="${coord:hours(1)}"
> start="2013-11-04T00:00Z"
> end="2013-11-04T24:00Z"
> timezone="UTC" xmlns="uri:oozie:coordinator:0.1">
>
> <action>
> <workflow>
> .
> .
> .
> </workflow>
> </action>
></coordinator-app>
>
>then workflow will run from 2013-11-04T00:00Z to 2013-11-04T12:00Z
>until it catches up (i.e., 12 or 13 times. Not sure as this is an
>example) to the current time. It will then follow the schedule. I would
>like it to just execute the next time the schedule is satisfied (i.e.,
>is
>2013-11-04T13:00Z)
>
>Thanks,
>Richard.
RE: Stopping Oozie Coordinator from running in the past
Posted by "Ross, Richard (Truven Health)" <ri...@truvenhealth.com>.
Hey Mona:
Thanks for your reply, and I agree that this should be parameterized (this was just an example). But I still have to make sure that the value of the parameters is correct and not in the past. In my mind I have just pushed the problem to another location. What I want to know is if there is a setting for the Coordinator that will make it so that I don't have to worry about this.
Thanks,
Richard.
-----Original Message-----
From: Mona Chitnis [mailto:chitnis@yahoo-inc.com]
Sent: Monday, November 04, 2013 1:47 PM
To: user@oozie.apache.org
Subject: Re: Stopping Oozie Coordinator from running in the past
Hi Richard,
Why not parameterize the coordinator "start" time? This start time should be when you wish the first instance to run.
E.g. <coordinator-app name="test-coord"
frequency="${coord:hours(1)}"
start=${my_start_time}
end="2013-11-04T24:00Z">
--
Mona
On 11/4/13 10:24 AM, "Ross, Richard (Truven Health)"
<ri...@truvenhealth.com> wrote:
>Forgot the Oozie version info: Oozie 3.3.2.
>
>Thanks,
>Richard.
>
>-----Original Message-----
>From: Ross, Richard (Truven Health)
>[mailto:richard.ross@truvenhealth.com]
>Sent: Monday, November 04, 2013 1:22 PM
>To: user@oozie.apache.org
>Subject: Stopping Oozie Coordinator from running in the past
>
>Hello:
>
>I have been playing around with Oozie Coordinator and noticed that when
>a coordinator definition is submitted, if the start value is in the
>past, it will run the workflow based on this past start and frequency
>values to "catch up" to the current time. Is there a way to tell Oozie
>to not run any workflows from the past?
>
>For example, if the actual time when the Coordinator definition is
>submitted is 2013-11-04T12:30Z and the coordinator.xml is:
>
><coordinator-app name="test-coord"
> frequency="${coord:hours(1)}"
> start="2013-11-04T00:00Z"
> end="2013-11-04T24:00Z"
> timezone="UTC" xmlns="uri:oozie:coordinator:0.1">
>
> <action>
> <workflow>
> .
> .
> .
> </workflow>
> </action>
></coordinator-app>
>
>then workflow will run from 2013-11-04T00:00Z to 2013-11-04T12:00Z
>until it catches up (i.e., 12 or 13 times. Not sure as this is an
>example) to the current time. It will then follow the schedule. I would
>like it to just execute the next time the schedule is satisfied (i.e.,
>is
>2013-11-04T13:00Z)
>
>Thanks,
>Richard.
Re: Stopping Oozie Coordinator from running in the past
Posted by Mona Chitnis <ch...@yahoo-inc.com>.
Hi Richard,
Why not parameterize the coordinator "start" time? This start time should
be when you wish the first instance to run.
E.g. <coordinator-app name="test-coord"
frequency="${coord:hours(1)}"
start=${my_start_time}
end="2013-11-04T24:00Z">
--
Mona
On 11/4/13 10:24 AM, "Ross, Richard (Truven Health)"
<ri...@truvenhealth.com> wrote:
>Forgot the Oozie version info: Oozie 3.3.2.
>
>Thanks,
>Richard.
>
>-----Original Message-----
>From: Ross, Richard (Truven Health)
>[mailto:richard.ross@truvenhealth.com]
>Sent: Monday, November 04, 2013 1:22 PM
>To: user@oozie.apache.org
>Subject: Stopping Oozie Coordinator from running in the past
>
>Hello:
>
>I have been playing around with Oozie Coordinator and noticed that when a
>coordinator definition is submitted, if the start value is in the past,
>it will run the workflow based on this past start and frequency values to
>"catch up" to the current time. Is there a way to tell Oozie to not run
>any workflows from the past?
>
>For example, if the actual time when the Coordinator definition is
>submitted is 2013-11-04T12:30Z and the coordinator.xml is:
>
><coordinator-app name="test-coord"
> frequency="${coord:hours(1)}"
> start="2013-11-04T00:00Z"
> end="2013-11-04T24:00Z"
> timezone="UTC" xmlns="uri:oozie:coordinator:0.1">
>
> <action>
> <workflow>
> .
> .
> .
> </workflow>
> </action>
></coordinator-app>
>
>then workflow will run from 2013-11-04T00:00Z to 2013-11-04T12:00Z until
>it catches up (i.e., 12 or 13 times. Not sure as this is an example) to
>the current time. It will then follow the schedule. I would like it to
>just execute the next time the schedule is satisfied (i.e., is
>2013-11-04T13:00Z)
>
>Thanks,
>Richard.
RE: Stopping Oozie Coordinator from running in the past
Posted by "Ross, Richard (Truven Health)" <ri...@truvenhealth.com>.
Forgot the Oozie version info: Oozie 3.3.2.
Thanks,
Richard.
-----Original Message-----
From: Ross, Richard (Truven Health) [mailto:richard.ross@truvenhealth.com]
Sent: Monday, November 04, 2013 1:22 PM
To: user@oozie.apache.org
Subject: Stopping Oozie Coordinator from running in the past
Hello:
I have been playing around with Oozie Coordinator and noticed that when a coordinator definition is submitted, if the start value is in the past, it will run the workflow based on this past start and frequency values to "catch up" to the current time. Is there a way to tell Oozie to not run any workflows from the past?
For example, if the actual time when the Coordinator definition is submitted is 2013-11-04T12:30Z and the coordinator.xml is:
<coordinator-app name="test-coord"
frequency="${coord:hours(1)}"
start="2013-11-04T00:00Z"
end="2013-11-04T24:00Z"
timezone="UTC" xmlns="uri:oozie:coordinator:0.1">
<action>
<workflow>
.
.
.
</workflow>
</action>
</coordinator-app>
then workflow will run from 2013-11-04T00:00Z to 2013-11-04T12:00Z until it catches up (i.e., 12 or 13 times. Not sure as this is an example) to the current time. It will then follow the schedule. I would like it to just execute the next time the schedule is satisfied (i.e., is 2013-11-04T13:00Z)
Thanks,
Richard.