You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by "J. McConnell" <j...@ubermenschconsulting.com> on 2014/11/28 23:05:09 UTC
start-/end-instance verification for OFFSET-specified ranges
I am either missing something or have hit a bug and would love if someone
could help me determine which it is. I am trying to specify start/end
instances for a month's worth of hourly data in a coordinator.xml file.
Perhaps there is a better way of doing this, but what I was attempting was:
<input-events>
<data-in name="requests" dataset="request-logs">
<start-instance>${coord:offset(-1, 'MONTH')}</start-instance>
<end-instance>${coord:offset(-1, 'HOUR')}</end-instance>
</data-in>
</input-events>
Where nominal time for an action would be midnight of the first of the next
month. When submitting this, I get the error:
start-instance should be equal or earlier than the end-instance
That doesn't seem right. Looking at the code where the error is generated,
this doesn't look right, either:
if (funcType == OFFSET) {
TimeUnit startU = TimeUnit.valueOf(startRestArg);
TimeUnit endU = TimeUnit.valueOf(endRestArg);
if (startU.getCalendarUnit() * startIndex >
endU.getCalendarUnit() * endIndex) {
throw new CommandException(ErrorCode.E1010,
" start-instance should be equal or
earlier than the end-instance \n"
+ XmlUtils.prettyPrint(event));
}
...
}
IIUC, the ints returned by startU/endU.getCalendarUnit() have no relation
to the magnitude of the unit, so scaling them by startIndex/endIndex is
meaningless. Just after this block, the actual dates are resolved. At that
point, it could be determined easily whether one instance came before
another.
Have I missed something? Does anyone have any suggestions for another
method of grabbing a month's worth of hourly data?
Thanks in advance for whatever help you can provide,
- J.
--
J. McConnell
Founder, Übermensch Consulting
Re: start-/end-instance verification for OFFSET-specified ranges
Posted by "J. McConnell" <j...@ubermenschconsulting.com>.
Thanks, Shwetha, very helpful!
Just created OOZIE-2084 and OOZIE-2085.
- J.
On Tue, Dec 2, 2014 at 12:11 AM, Shwetha GS <sh...@inmobi.com> wrote:
> Yes, that looks like a bug. Can you please file a jira with these details?
>
> Its possible to extend oozie EL functions and provide a custom EL function.
> Sorry, I couldn't find the documentation. You can file a jira for that as
> well.
>
> In Apache falcon project, we have extended these EL functions and provided
> EL functions like:
> 1. currentMonth(day, hr, min) - current month resolves to the beginning of
> current month with respect to nominal time. day, hr and min are offsets
> with respect to the resolved time
> 2. lastMonth(day, hour, min) - last month resolves to the beginning of
> previous month with respect to nominal time. day, hr and min are offsets
> with respect to the resolved time
>
> For your usecase, you can use start as lastMonth(0, 0, 0) and end as
> currentMonth(0, -1, 0). These start and end functions need not be changed
> even if you change the nominal time to any other time in the same month.
> The code for this is at
> https://github.com/apache/incubator-falcon/tree/master/oozie-el-extensions
> .
> You can either use this or use it as a reference to write your own EL
> functions. You need to include the EL implementation jar in oozie
> server(libext) and oozie-site.xml should have the corresponding configs.
>
> -Shwetha
>
>
>
> On Mon, Dec 1, 2014 at 9:59 PM, J. McConnell <j...@ubermenschconsulting.com>
> wrote:
>
> > On Sun, Nov 30, 2014 at 6:54 AM, Idris Ali <ps...@gmail.com> wrote:
> >
> > >
> > > If the intention is to read hourly data for a month, then the simplest
> > way
> > > is to specify like this.
> > >
> > > <data-in name="input" dataset="logs">
> > > <start-instance>${coord:current( -(coord:daysInMonth(0) -
> > > 1)*24 )}</start-instance>
> > > <end-instance>${coord:current(0)}</end-instance>
> > > </data-in>
> > >
> > > This takes care of any month, however special handling is required for
> > > daylight saving timezone, for details refer the coordinator functional
> > spec
> > > here.
> >
> >
> > Yes, that is indeed the intention and this was my first thought. However,
> > it is the special handling for daylight saving time that I am
> specifically
> > looking to avoid. I don't see anything in the coordinator functional spec
> > that will help with this. The coord:hoursInDay() function is great, but
> > what is needed in this case is a coord:hoursInMonth() function.
> >
> > That said, I thought the approach of using the coord:offset() function
> > would be a good way of doing this, but I've hit what appears to be a bug.
> > Can anyone confirm that this is indeed a bug? Does anyone know of a good
> > workaround, either with coord:daysInMonth()/hoursInDay() or some other
> way?
> > Is it possible to extend the EL language locally to support an
> > hoursInMonth() method?
> >
> > Any help would be appreciated. Thanks,
> >
> > - J.
> >
>
> --
> _____________________________________________________________
> The information contained in this communication is intended solely for the
> use of the individual or entity to whom it is addressed and others
> authorized to receive it. It may contain confidential or legally privileged
> information. If you are not the intended recipient you are hereby notified
> that any disclosure, copying, distribution or taking any action in reliance
> on the contents of this information is strictly prohibited and may be
> unlawful. If you have received this communication in error, please notify
> us immediately by responding to this email and then delete it from your
> system. The firm is neither liable for the proper and complete transmission
> of the information contained in this communication nor for any delay in its
> receipt.
>
--
J. McConnell
Founder, Übermensch Consulting
Re: start-/end-instance verification for OFFSET-specified ranges
Posted by Shwetha GS <sh...@inmobi.com>.
Yes, that looks like a bug. Can you please file a jira with these details?
Its possible to extend oozie EL functions and provide a custom EL function.
Sorry, I couldn't find the documentation. You can file a jira for that as
well.
In Apache falcon project, we have extended these EL functions and provided
EL functions like:
1. currentMonth(day, hr, min) - current month resolves to the beginning of
current month with respect to nominal time. day, hr and min are offsets
with respect to the resolved time
2. lastMonth(day, hour, min) - last month resolves to the beginning of
previous month with respect to nominal time. day, hr and min are offsets
with respect to the resolved time
For your usecase, you can use start as lastMonth(0, 0, 0) and end as
currentMonth(0, -1, 0). These start and end functions need not be changed
even if you change the nominal time to any other time in the same month.
The code for this is at
https://github.com/apache/incubator-falcon/tree/master/oozie-el-extensions.
You can either use this or use it as a reference to write your own EL
functions. You need to include the EL implementation jar in oozie
server(libext) and oozie-site.xml should have the corresponding configs.
-Shwetha
On Mon, Dec 1, 2014 at 9:59 PM, J. McConnell <j...@ubermenschconsulting.com>
wrote:
> On Sun, Nov 30, 2014 at 6:54 AM, Idris Ali <ps...@gmail.com> wrote:
>
> >
> > If the intention is to read hourly data for a month, then the simplest
> way
> > is to specify like this.
> >
> > <data-in name="input" dataset="logs">
> > <start-instance>${coord:current( -(coord:daysInMonth(0) -
> > 1)*24 )}</start-instance>
> > <end-instance>${coord:current(0)}</end-instance>
> > </data-in>
> >
> > This takes care of any month, however special handling is required for
> > daylight saving timezone, for details refer the coordinator functional
> spec
> > here.
>
>
> Yes, that is indeed the intention and this was my first thought. However,
> it is the special handling for daylight saving time that I am specifically
> looking to avoid. I don't see anything in the coordinator functional spec
> that will help with this. The coord:hoursInDay() function is great, but
> what is needed in this case is a coord:hoursInMonth() function.
>
> That said, I thought the approach of using the coord:offset() function
> would be a good way of doing this, but I've hit what appears to be a bug.
> Can anyone confirm that this is indeed a bug? Does anyone know of a good
> workaround, either with coord:daysInMonth()/hoursInDay() or some other way?
> Is it possible to extend the EL language locally to support an
> hoursInMonth() method?
>
> Any help would be appreciated. Thanks,
>
> - J.
>
--
_____________________________________________________________
The information contained in this communication is intended solely for the
use of the individual or entity to whom it is addressed and others
authorized to receive it. It may contain confidential or legally privileged
information. If you are not the intended recipient you are hereby notified
that any disclosure, copying, distribution or taking any action in reliance
on the contents of this information is strictly prohibited and may be
unlawful. If you have received this communication in error, please notify
us immediately by responding to this email and then delete it from your
system. The firm is neither liable for the proper and complete transmission
of the information contained in this communication nor for any delay in its
receipt.
Re: start-/end-instance verification for OFFSET-specified ranges
Posted by "J. McConnell" <j...@ubermenschconsulting.com>.
On Sun, Nov 30, 2014 at 6:54 AM, Idris Ali <ps...@gmail.com> wrote:
>
> If the intention is to read hourly data for a month, then the simplest way
> is to specify like this.
>
> <data-in name="input" dataset="logs">
> <start-instance>${coord:current( -(coord:daysInMonth(0) -
> 1)*24 )}</start-instance>
> <end-instance>${coord:current(0)}</end-instance>
> </data-in>
>
> This takes care of any month, however special handling is required for
> daylight saving timezone, for details refer the coordinator functional spec
> here.
Yes, that is indeed the intention and this was my first thought. However,
it is the special handling for daylight saving time that I am specifically
looking to avoid. I don't see anything in the coordinator functional spec
that will help with this. The coord:hoursInDay() function is great, but
what is needed in this case is a coord:hoursInMonth() function.
That said, I thought the approach of using the coord:offset() function
would be a good way of doing this, but I've hit what appears to be a bug.
Can anyone confirm that this is indeed a bug? Does anyone know of a good
workaround, either with coord:daysInMonth()/hoursInDay() or some other way?
Is it possible to extend the EL language locally to support an
hoursInMonth() method?
Any help would be appreciated. Thanks,
- J.
Re: start-/end-instance verification for OFFSET-specified ranges
Posted by Idris Ali <ps...@gmail.com>.
Hi,
If the intention is to read hourly data for a month, then the simplest way
is to specify like this.
<data-in name="input" dataset="logs">
<start-instance>${coord:current( -(coord:daysInMonth(0) -
1)*24 )}</start-instance>
<end-instance>${coord:current(0)}</end-instance>
</data-in>
This takes care of any month, however special handling is required for
daylight saving timezone, for details refer the coordinator functional spec
here.
https://oozie.apache.org/docs/3.1.3-incubating/CoordinatorFunctionalSpec.html
Thanks,
-Idris
On Sat, Nov 29, 2014 at 3:35 AM, J. McConnell <j...@ubermenschconsulting.com>
wrote:
> I am either missing something or have hit a bug and would love if someone
> could help me determine which it is. I am trying to specify start/end
> instances for a month's worth of hourly data in a coordinator.xml file.
> Perhaps there is a better way of doing this, but what I was attempting was:
>
> <input-events>
> <data-in name="requests" dataset="request-logs">
> <start-instance>${coord:offset(-1, 'MONTH')}</start-instance>
> <end-instance>${coord:offset(-1, 'HOUR')}</end-instance>
> </data-in>
> </input-events>
>
> Where nominal time for an action would be midnight of the first of the next
> month. When submitting this, I get the error:
>
> start-instance should be equal or earlier than the end-instance
>
> That doesn't seem right. Looking at the code where the error is generated,
> this doesn't look right, either:
>
> if (funcType == OFFSET) {
> TimeUnit startU = TimeUnit.valueOf(startRestArg);
> TimeUnit endU = TimeUnit.valueOf(endRestArg);
> if (startU.getCalendarUnit() * startIndex >
> endU.getCalendarUnit() * endIndex) {
> throw new CommandException(ErrorCode.E1010,
> " start-instance should be equal or
> earlier than the end-instance \n"
> + XmlUtils.prettyPrint(event));
> }
> ...
> }
>
> IIUC, the ints returned by startU/endU.getCalendarUnit() have no relation
> to the magnitude of the unit, so scaling them by startIndex/endIndex is
> meaningless. Just after this block, the actual dates are resolved. At that
> point, it could be determined easily whether one instance came before
> another.
>
> Have I missed something? Does anyone have any suggestions for another
> method of grabbing a month's worth of hourly data?
>
> Thanks in advance for whatever help you can provide,
>
> - J.
>
> --
> J. McConnell
> Founder, Übermensch Consulting
>