You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by "J. McConnell" <j...@ubermenschconsulting.com> on 2014/12/01 17:29:08 UTC
Re: start-/end-instance verification for OFFSET-specified ranges
On Sun, Nov 30, 2014 at 6:54 AM, Idris Ali <ps...@gmail.com> wrote:
>
> If the intention is to read hourly data for a month, then the simplest way
> is to specify like this.
>
> <data-in name="input" dataset="logs">
> <start-instance>${coord:current( -(coord:daysInMonth(0) -
> 1)*24 )}</start-instance>
> <end-instance>${coord:current(0)}</end-instance>
> </data-in>
>
> This takes care of any month, however special handling is required for
> daylight saving timezone, for details refer the coordinator functional spec
> here.
Yes, that is indeed the intention and this was my first thought. However,
it is the special handling for daylight saving time that I am specifically
looking to avoid. I don't see anything in the coordinator functional spec
that will help with this. The coord:hoursInDay() function is great, but
what is needed in this case is a coord:hoursInMonth() function.
That said, I thought the approach of using the coord:offset() function
would be a good way of doing this, but I've hit what appears to be a bug.
Can anyone confirm that this is indeed a bug? Does anyone know of a good
workaround, either with coord:daysInMonth()/hoursInDay() or some other way?
Is it possible to extend the EL language locally to support an
hoursInMonth() method?
Any help would be appreciated. Thanks,
- J.
Re: start-/end-instance verification for OFFSET-specified ranges
Posted by "J. McConnell" <j...@ubermenschconsulting.com>.
Thanks, Shwetha, very helpful!
Just created OOZIE-2084 and OOZIE-2085.
- J.
On Tue, Dec 2, 2014 at 12:11 AM, Shwetha GS <sh...@inmobi.com> wrote:
> Yes, that looks like a bug. Can you please file a jira with these details?
>
> Its possible to extend oozie EL functions and provide a custom EL function.
> Sorry, I couldn't find the documentation. You can file a jira for that as
> well.
>
> In Apache falcon project, we have extended these EL functions and provided
> EL functions like:
> 1. currentMonth(day, hr, min) - current month resolves to the beginning of
> current month with respect to nominal time. day, hr and min are offsets
> with respect to the resolved time
> 2. lastMonth(day, hour, min) - last month resolves to the beginning of
> previous month with respect to nominal time. day, hr and min are offsets
> with respect to the resolved time
>
> For your usecase, you can use start as lastMonth(0, 0, 0) and end as
> currentMonth(0, -1, 0). These start and end functions need not be changed
> even if you change the nominal time to any other time in the same month.
> The code for this is at
> https://github.com/apache/incubator-falcon/tree/master/oozie-el-extensions
> .
> You can either use this or use it as a reference to write your own EL
> functions. You need to include the EL implementation jar in oozie
> server(libext) and oozie-site.xml should have the corresponding configs.
>
> -Shwetha
>
>
>
> On Mon, Dec 1, 2014 at 9:59 PM, J. McConnell <j...@ubermenschconsulting.com>
> wrote:
>
> > On Sun, Nov 30, 2014 at 6:54 AM, Idris Ali <ps...@gmail.com> wrote:
> >
> > >
> > > If the intention is to read hourly data for a month, then the simplest
> > way
> > > is to specify like this.
> > >
> > > <data-in name="input" dataset="logs">
> > > <start-instance>${coord:current( -(coord:daysInMonth(0) -
> > > 1)*24 )}</start-instance>
> > > <end-instance>${coord:current(0)}</end-instance>
> > > </data-in>
> > >
> > > This takes care of any month, however special handling is required for
> > > daylight saving timezone, for details refer the coordinator functional
> > spec
> > > here.
> >
> >
> > Yes, that is indeed the intention and this was my first thought. However,
> > it is the special handling for daylight saving time that I am
> specifically
> > looking to avoid. I don't see anything in the coordinator functional spec
> > that will help with this. The coord:hoursInDay() function is great, but
> > what is needed in this case is a coord:hoursInMonth() function.
> >
> > That said, I thought the approach of using the coord:offset() function
> > would be a good way of doing this, but I've hit what appears to be a bug.
> > Can anyone confirm that this is indeed a bug? Does anyone know of a good
> > workaround, either with coord:daysInMonth()/hoursInDay() or some other
> way?
> > Is it possible to extend the EL language locally to support an
> > hoursInMonth() method?
> >
> > Any help would be appreciated. Thanks,
> >
> > - J.
> >
>
> --
> _____________________________________________________________
> The information contained in this communication is intended solely for the
> use of the individual or entity to whom it is addressed and others
> authorized to receive it. It may contain confidential or legally privileged
> information. If you are not the intended recipient you are hereby notified
> that any disclosure, copying, distribution or taking any action in reliance
> on the contents of this information is strictly prohibited and may be
> unlawful. If you have received this communication in error, please notify
> us immediately by responding to this email and then delete it from your
> system. The firm is neither liable for the proper and complete transmission
> of the information contained in this communication nor for any delay in its
> receipt.
>
--
J. McConnell
Founder, Übermensch Consulting
Re: start-/end-instance verification for OFFSET-specified ranges
Posted by Shwetha GS <sh...@inmobi.com>.
Yes, that looks like a bug. Can you please file a jira with these details?
Its possible to extend oozie EL functions and provide a custom EL function.
Sorry, I couldn't find the documentation. You can file a jira for that as
well.
In Apache falcon project, we have extended these EL functions and provided
EL functions like:
1. currentMonth(day, hr, min) - current month resolves to the beginning of
current month with respect to nominal time. day, hr and min are offsets
with respect to the resolved time
2. lastMonth(day, hour, min) - last month resolves to the beginning of
previous month with respect to nominal time. day, hr and min are offsets
with respect to the resolved time
For your usecase, you can use start as lastMonth(0, 0, 0) and end as
currentMonth(0, -1, 0). These start and end functions need not be changed
even if you change the nominal time to any other time in the same month.
The code for this is at
https://github.com/apache/incubator-falcon/tree/master/oozie-el-extensions.
You can either use this or use it as a reference to write your own EL
functions. You need to include the EL implementation jar in oozie
server(libext) and oozie-site.xml should have the corresponding configs.
-Shwetha
On Mon, Dec 1, 2014 at 9:59 PM, J. McConnell <j...@ubermenschconsulting.com>
wrote:
> On Sun, Nov 30, 2014 at 6:54 AM, Idris Ali <ps...@gmail.com> wrote:
>
> >
> > If the intention is to read hourly data for a month, then the simplest
> way
> > is to specify like this.
> >
> > <data-in name="input" dataset="logs">
> > <start-instance>${coord:current( -(coord:daysInMonth(0) -
> > 1)*24 )}</start-instance>
> > <end-instance>${coord:current(0)}</end-instance>
> > </data-in>
> >
> > This takes care of any month, however special handling is required for
> > daylight saving timezone, for details refer the coordinator functional
> spec
> > here.
>
>
> Yes, that is indeed the intention and this was my first thought. However,
> it is the special handling for daylight saving time that I am specifically
> looking to avoid. I don't see anything in the coordinator functional spec
> that will help with this. The coord:hoursInDay() function is great, but
> what is needed in this case is a coord:hoursInMonth() function.
>
> That said, I thought the approach of using the coord:offset() function
> would be a good way of doing this, but I've hit what appears to be a bug.
> Can anyone confirm that this is indeed a bug? Does anyone know of a good
> workaround, either with coord:daysInMonth()/hoursInDay() or some other way?
> Is it possible to extend the EL language locally to support an
> hoursInMonth() method?
>
> Any help would be appreciated. Thanks,
>
> - J.
>
--
_____________________________________________________________
The information contained in this communication is intended solely for the
use of the individual or entity to whom it is addressed and others
authorized to receive it. It may contain confidential or legally privileged
information. If you are not the intended recipient you are hereby notified
that any disclosure, copying, distribution or taking any action in reliance
on the contents of this information is strictly prohibited and may be
unlawful. If you have received this communication in error, please notify
us immediately by responding to this email and then delete it from your
system. The firm is neither liable for the proper and complete transmission
of the information contained in this communication nor for any delay in its
receipt.