You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by Scott Preddy <sc...@gmail.com> on 2014/02/20 18:03:18 UTC

oozie 3.3 coordinator question

Will the snipped below over the same 23 logs it ran over the previous hour
(i.e. just bumping up
the log iterator by 1) each hour, or is oozie going to run the action once
24 logs are present, then not kick off the action again until 24 new logs
are present? I think it is the former, but just
making sure.

<coordinator-app name="app-coord-daily" frequency="${coord:days(1)}"
                    start="2009-01-01T24:00Z" end="2009-12-31T24:00Z"
timezone="UTC"
                    xmlns="uri:oozie:coordinator:0.1">
      <datasets>

<include>hdfs://foo:8020/app/dataset-definitions/datasets.xml</include>
      </datasets>
      <input-events>
        <data-in name="input" dataset="1HourLogs">
          <start-instance>${coord:current(-23)}</start-instance>
          <end-instance>${coord:current(0)}</end-instance>
        </data-in>
      </input-events>
      <output-events>
        <data-out name="output" dataset="1DayLogs">
          <instance>${coord:current(0)}</instance>
        </data-out>
      </output-events>
      <action>
        <workflow>
        ...
       </workflow>
      </action>
   </coordinator-app>

Re: oozie 3.3 coordinator question

Posted by Scott Preddy <sc...@gmail.com>.
Actually, my confusion was here (just answered my own question).

 <coordinator-app name="app-coord-daily" frequency="${coord:days(1)}"
                    start="2009-01-01T24:00Z" end="2009-12-31T24:00Z"
timezone="UTC"
                    xmlns="uri:oozie:coordinator:0.1">

If the logs are continually present, having the coordinator run once a day
will make it so, 24 new logs are grabbed each time.

Thanks.


On Thu, Feb 20, 2014 at 11:21 AM, Purshotam Shah <pu...@yahoo-inc.com>wrote:

>
> Yes. If you dataset ³1HourLogs² is hourly, then every time it going to
> look for 23 previous hour logs  + 1 .
>
>
>
> On 2/20/14, 9:03 AM, "Scott Preddy" <sc...@gmail.com> wrote:
>
> >Will the snipped below over the same 23 logs it ran over the previous hour
> >(i.e. just bumping up
> >the log iterator by 1) each hour, or is oozie going to run the action once
> >24 logs are present, then not kick off the action again until 24 new logs
> >are present? I think it is the former, but just
> >making sure.
> >
> ><coordinator-app name="app-coord-daily" frequency="${coord:days(1)}"
> >                    start="2009-01-01T24:00Z" end="2009-12-31T24:00Z"
> >timezone="UTC"
> >                    xmlns="uri:oozie:coordinator:0.1">
> >      <datasets>
> >
> ><include>hdfs://foo:8020/app/dataset-definitions/datasets.xml</include>
> >      </datasets>
> >      <input-events>
> >        <data-in name="input" dataset="1HourLogs">
> >          <start-instance>${coord:current(-23)}</start-instance>
> >          <end-instance>${coord:current(0)}</end-instance>
> >        </data-in>
> >      </input-events>
> >      <output-events>
> >        <data-out name="output" dataset="1DayLogs">
> >          <instance>${coord:current(0)}</instance>
> >        </data-out>
> >      </output-events>
> >      <action>
> >        <workflow>
> >        ...
> >       </workflow>
> >      </action>
> >   </coordinator-app>
>
>

Re: oozie 3.3 coordinator question

Posted by Purshotam Shah <pu...@yahoo-inc.com>.
Yes. If you dataset ³1HourLogs² is hourly, then every time it going to
look for 23 previous hour logs  + 1 .



On 2/20/14, 9:03 AM, "Scott Preddy" <sc...@gmail.com> wrote:

>Will the snipped below over the same 23 logs it ran over the previous hour
>(i.e. just bumping up
>the log iterator by 1) each hour, or is oozie going to run the action once
>24 logs are present, then not kick off the action again until 24 new logs
>are present? I think it is the former, but just
>making sure.
>
><coordinator-app name="app-coord-daily" frequency="${coord:days(1)}"
>                    start="2009-01-01T24:00Z" end="2009-12-31T24:00Z"
>timezone="UTC"
>                    xmlns="uri:oozie:coordinator:0.1">
>      <datasets>
>
><include>hdfs://foo:8020/app/dataset-definitions/datasets.xml</include>
>      </datasets>
>      <input-events>
>        <data-in name="input" dataset="1HourLogs">
>          <start-instance>${coord:current(-23)}</start-instance>
>          <end-instance>${coord:current(0)}</end-instance>
>        </data-in>
>      </input-events>
>      <output-events>
>        <data-out name="output" dataset="1DayLogs">
>          <instance>${coord:current(0)}</instance>
>        </data-out>
>      </output-events>
>      <action>
>        <workflow>
>        ...
>       </workflow>
>      </action>
>   </coordinator-app>