You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by Eduardo Afonso Ferreira <ea...@yahoo.com> on 2012/11/14 20:28:44 UTC
Creating/implementing custom EL Functions.
Hey,
Is it possible to implement custom EL Functions without having to change Oozie source code?
I did not find documentation about how to implement custom EL Functions.
Would someone throw some links my way about where I can find information on this?
Thank you.
Eduardo.
Re: Creating/implementing custom EL Functions.
Posted by Eduardo Afonso Ferreira <ea...@yahoo.com>.
Sure, I'll do it :)
________________________________
From: Mohammad Islam <mi...@yahoo.com>
To: "user@oozie.apache.org" <us...@oozie.apache.org>; Eduardo Afonso Ferreira <ea...@yahoo.com>
Sent: Saturday, November 17, 2012 1:55 AM
Subject: Re: Creating/implementing custom EL Functions.
Good to hear that it worked.
Whenever got time, can you please write & upload a document on "how to write a custom EL functions" before you forget.
Will be really helpful for others.
Regards,
Mohammad
________________________________
From: Eduardo Afonso Ferreira <ea...@yahoo.com>
To: "user@oozie.apache.org" <us...@oozie.apache.org>; Mohammad Islam <mi...@yahoo.com>
Sent: Friday, November 16, 2012 5:17 PM
Subject: Re: Creating/implementing custom EL Functions.
Ah, I got it now. And got it to work.
Thank you :)
________________________________
From: Mohammad Islam <mi...@yahoo.com>
To: "user@oozie.apache.org" <us...@oozie.apache.org>; Eduardo Afonso Ferreira <ea...@yahoo.com>
Sent: Thursday, November 15, 2012 6:31 PM
Subject: Re: Creating/implementing custom EL Functions.
Can you please check the "echo" method that I mentioned?
Please check the code at:
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java
and search for "ph1_coord_formatTime_echo". Also check how it is defined in ooze-default.xml.
That means you need to follow the same pattern that we used for formatTime(...)
________________________________
From: Eduardo Afonso Ferreira <ea...@yahoo.com>
To: "user@oozie.apache.org" <us...@oozie.apache.org>
Sent: Thursday, November 15, 2012 2:46 PM
Subject: Re: Creating/implementing custom EL Functions.
Hey,
I added those three but the result appears to be the same.
Here's the stack trace I see in oozie.log. It looks like it does not resolve coord:nominalTime() but just passes that as a string to the function call.
Notice the "Caused by" towards the end of the stack trace:
.....
Caused by: java.text.ParseException: Unparseable date: "coord:nominalTime()"
.....
Here's the stack trace:
2012-11-15 17:40:12,213 WARN V1JobsServlet:542 - USER[aspen] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] URL[POST http://localhost:11000/oozie/v1/jobs] error[E1004], E1004: Expression language evaluation error [U
nable to evaluate :${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}:
], java.lang.Exception: Unable to evaluate :${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}:
org.apache.oozie.servlet.XServletException: E1004: Expression language evaluation error [Unable to evaluate :${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}:
], java.lang.Exception: Unable to evaluate :${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}:
at org.apache.oozie.servlet.V1JobsServlet.submitCoordinatorJob(V1JobsServlet.java:205)
at org.apache.oozie.servlet.V1JobsServlet.submitJob(V1JobsServlet.java:80)
at org.apache.oozie.servlet.BaseJobsServlet.doPost(BaseJobsServlet.java:95)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
at org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:285)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.oozie.servlet.AuthFilter$2.doFilter(AuthFilter.java:126)
at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:372)
at org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:131)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.oozie.servlet.HostnameFilter.doFilter(HostnameFilter.java:67)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.oozie.CoordinatorEngineException: E1004: Expression language evaluation error [Unable to evaluate :${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}:
], java.lang.Exception: Unable to evaluate :${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}:
at org.apache.oozie.CoordinatorEngine.submitJob(CoordinatorEngine.java:374)
at org.apache.oozie.servlet.V1JobsServlet.submitCoordinatorJob(V1JobsServlet.java:200)
... 25 more
Caused by: org.apache.oozie.command.CommandException: E1004: Expression language evaluation error [Unable to evaluate :${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}:
], java.lang.Exception: Unable to evaluate :${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}:
at org.apache.oozie.command.coord.CoordSubmitXCommand.submit(CoordSubmitXCommand.java:272)
at org.apache.oozie.command.SubmitTransitionXCommand.execute(SubmitTransitionXCommand.java:81)
at org.apache.oozie.command.SubmitTransitionXCommand.execute(SubmitTransitionXCommand.java:29)
at org.apache.oozie.command.XCommand.call(XCommand.java:277)
at org.apache.oozie.CoordinatorEngine.submitJob(CoordinatorEngine.java:371)
... 26 more
Caused by: org.apache.oozie.coord.CoordinatorJobException: E1004: Expression language evaluation error [Unable to evaluate :${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}:
], java.lang.Exception: Unable to evaluate :${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}:
at org.apache.oozie.command.coord.CoordSubmitXCommand.resolveTagContents(CoordSubmitXCommand.java:823)
at org.apache.oozie.command.coord.CoordSubmitXCommand.resolveInitial(CoordSubmitXCommand.java:663)
at org.apache.oozie.command.coord.CoordSubmitXCommand.basicResolveAndIncludeDS(CoordSubmitXCommand.java:484)
at org.apache.oozie.command.coord.CoordSubmitXCommand.submit(CoordSubmitXCommand.java:222)
... 30 more
Caused by: java.lang.Exception: Unable to evaluate :${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}:
at org.apache.oozie.coord.CoordELFunctions.evalAndWrap(CoordELFunctions.java:551)
at org.apache.oozie.command.coord.CoordSubmitXCommand.resolveTagContents(CoordSubmitXCommand.java:819)
... 33 more
Caused by: java.text.ParseException: Unparseable date: "coord:nominalTime()"
at java.text.DateFormat.parse(DateFormat.java:337)
at com.turner.util.Utils.formatTimeTZ(Unknown Source)
at com.turner.util.Utils.formatTimeTZ(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.commons.el.FunctionInvocation.evaluate(FunctionInvocation.java:172)
at org.apache.commons.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:263)
at org.apache.commons.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:190)
at org.apache.oozie.util.ELEvaluator.evaluate(ELEvaluator.java:203)
at org.apache.oozie.coord.CoordELFunctions.evalAndWrap(CoordELFunctions.java:542)
... 34 more
________________________________
From: Mohammad Islam <mi...@yahoo.com>
To: "user@oozie.apache.org" <us...@oozie.apache.org>; Eduardo Afonso Ferreira <ea...@yahoo.com>
Sent: Thursday, November 15, 2012 4:45 PM
Subject: Re: Creating/implementing custom EL Functions.
Hi eduardo,
You have to add the similar thing in three other properties. Please search for "formatTime" in ooze-default.xml. Coordinator EL functions are evaluated in different contexts (multiple phases). Other cases, it called "*echo" method such as "coord_formatTime_echo". For example, during job submission value nominalTime is not defined.
<name>oozie.service.ELService.ext.functions.coord-action-create</name>
and
oozie.service.ELService.ext.functions.coord-action-create-inst
oozie.service.ELService.ext.functions.coord-action-start
Regards,
Mohammad
________________________________
From: Eduardo Afonso Ferreira <ea...@yahoo.com>
To: "user@oozie.apache.org" <us...@oozie.apache.org>
Sent: Thursday, November 15, 2012 1:10 PM
Subject: Re: Creating/implementing custom EL Functions.
Hey,
I'm trying to do some date manipulation via custom EL functions.
I added a function coord:formatTimeTZ() very similar to the original coord:formatTime() but the new one will take an offset and optionally a timezone.
Here's an example of usage (below).
The problem I see now is when I use only the original coord functions, all functions are called and processed fine, but when I use my new function (formatTimeTZ), the attribute coord:nominalTime() is not called, but it simply passes the string "coord:nominalTime()" to the function.
Did I miss anything?
<property>
<name>START_ROW_DT</name>
<value>${coord:dateOffset(coord:nominalTime(), -FREQUENCY-OFFSET, 'MINUTE')}</value>
</property>
<property>
<name>START_ROW_FORM_1</name>
<value>${coord:formatTime(coord:dateOffset(coord:nominalTime(), -FREQUENCY-OFFSET, 'MINUTE'), "'min-'yyyyMMddHHmm")}</value>
</property>
<property>
<name>START_ROW_FORM_2</name>
<value>${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}</value>
</property>
NOTE: Here's what I added to oozie-site.xml to define the new function. I also added my new jar to oozie.war with oozie setup.
<property>
<name>oozie.service.ELService.ext.functions.coord-job-submit-data</name>
<value>
coord:formatTimeTZ=com.turner.util.Utils#formatTimeTZ
</value>
<description>
EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD.
</description>
</property>
________________________________
From: Harsh J <ha...@cloudera.com>
To: user@oozie.apache.org; Eduardo Afonso Ferreira <ea...@yahoo.com>
Sent: Wednesday, November 14, 2012 3:10 PM
Subject: Re: Creating/implementing custom EL Functions.
Hey Eduardo,
There is lack of docs in this area but you can certainly plug in
custom EL functions and give them aliases.
Look at all the configs prefixed
"oozie.service.ELService.ext.functions" in
http://oozie.apache.org/docs/3.2.0-incubating/oozie-default.xml. You
can create your own function that way by mapping a name to a class
plus method call. The class needs to be in the Oozie deployed WAR
though, so this isn't a dynamic thing. See the examples around these
config entries in the link for the inbuilt definitions we carry.
I'd also strongly recommend contributing back nifty EL extensions, so
that its not a maintenance pain in the long run for you guys - and
others can use it as well :)
On Thu, Nov 15, 2012 at 12:58 AM, Eduardo Afonso Ferreira
<ea...@yahoo.com> wrote:
> Hey,
>
> Is it possible to implement custom EL Functions without having to change Oozie source code?
> I did not find documentation about how to implement custom EL Functions.
> Would someone throw some links my way about where I can find information on this?
>
> Thank you.
> Eduardo.
--
Harsh J
Re: Creating/implementing custom EL Functions.
Posted by Mohammad Islam <mi...@yahoo.com>.
Good to hear that it worked.
Whenever got time, can you please write & upload a document on "how to write a custom EL functions" before you forget.
Will be really helpful for others.
Regards,
Mohammad
________________________________
From: Eduardo Afonso Ferreira <ea...@yahoo.com>
To: "user@oozie.apache.org" <us...@oozie.apache.org>; Mohammad Islam <mi...@yahoo.com>
Sent: Friday, November 16, 2012 5:17 PM
Subject: Re: Creating/implementing custom EL Functions.
Ah, I got it now. And got it to work.
Thank you :)
________________________________
From: Mohammad Islam <mi...@yahoo.com>
To: "user@oozie.apache.org" <us...@oozie.apache.org>; Eduardo Afonso Ferreira <ea...@yahoo.com>
Sent: Thursday, November 15, 2012 6:31 PM
Subject: Re: Creating/implementing custom EL Functions.
Can you please check the "echo" method that I mentioned?
Please check the code at:
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java
and search for "ph1_coord_formatTime_echo". Also check how it is defined in ooze-default.xml.
That means you need to follow the same pattern that we used for formatTime(...)
________________________________
From: Eduardo Afonso Ferreira <ea...@yahoo.com>
To: "user@oozie.apache.org" <us...@oozie.apache.org>
Sent: Thursday, November 15, 2012 2:46 PM
Subject: Re: Creating/implementing custom EL Functions.
Hey,
I added those three but the result appears to be the same.
Here's the stack trace I see in oozie.log. It looks like it does not resolve coord:nominalTime() but just passes that as a string to the function call.
Notice the "Caused by" towards the end of the stack trace:
.....
Caused by: java.text.ParseException: Unparseable date: "coord:nominalTime()"
.....
Here's the stack trace:
2012-11-15 17:40:12,213 WARN V1JobsServlet:542 - USER[aspen] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] URL[POST http://localhost:11000/oozie/v1/jobs] error[E1004], E1004: Expression language evaluation error [U
nable to evaluate :${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}:
], java.lang.Exception: Unable to evaluate :${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}:
org.apache.oozie.servlet.XServletException: E1004: Expression language evaluation error [Unable to evaluate :${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}:
], java.lang.Exception: Unable to evaluate :${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}:
at org.apache.oozie.servlet.V1JobsServlet.submitCoordinatorJob(V1JobsServlet.java:205)
at org.apache.oozie.servlet.V1JobsServlet.submitJob(V1JobsServlet.java:80)
at org.apache.oozie.servlet.BaseJobsServlet.doPost(BaseJobsServlet.java:95)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
at org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:285)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.oozie.servlet.AuthFilter$2.doFilter(AuthFilter.java:126)
at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:372)
at org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:131)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.oozie.servlet.HostnameFilter.doFilter(HostnameFilter.java:67)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.oozie.CoordinatorEngineException: E1004: Expression language evaluation error [Unable to evaluate :${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}:
], java.lang.Exception: Unable to evaluate :${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}:
at org.apache.oozie.CoordinatorEngine.submitJob(CoordinatorEngine.java:374)
at org.apache.oozie.servlet.V1JobsServlet.submitCoordinatorJob(V1JobsServlet.java:200)
... 25 more
Caused by: org.apache.oozie.command.CommandException: E1004: Expression language evaluation error [Unable to evaluate :${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}:
], java.lang.Exception: Unable to evaluate :${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}:
at org.apache.oozie.command.coord.CoordSubmitXCommand.submit(CoordSubmitXCommand.java:272)
at org.apache.oozie.command.SubmitTransitionXCommand.execute(SubmitTransitionXCommand.java:81)
at org.apache.oozie.command.SubmitTransitionXCommand.execute(SubmitTransitionXCommand.java:29)
at org.apache.oozie.command.XCommand.call(XCommand.java:277)
at org.apache.oozie.CoordinatorEngine.submitJob(CoordinatorEngine.java:371)
... 26 more
Caused by: org.apache.oozie.coord.CoordinatorJobException: E1004: Expression language evaluation error [Unable to evaluate :${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}:
], java.lang.Exception: Unable to evaluate :${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}:
at org.apache.oozie.command.coord.CoordSubmitXCommand.resolveTagContents(CoordSubmitXCommand.java:823)
at org.apache.oozie.command.coord.CoordSubmitXCommand.resolveInitial(CoordSubmitXCommand.java:663)
at org.apache.oozie.command.coord.CoordSubmitXCommand.basicResolveAndIncludeDS(CoordSubmitXCommand.java:484)
at org.apache.oozie.command.coord.CoordSubmitXCommand.submit(CoordSubmitXCommand.java:222)
... 30 more
Caused by: java.lang.Exception: Unable to evaluate :${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}:
at org.apache.oozie.coord.CoordELFunctions.evalAndWrap(CoordELFunctions.java:551)
at org.apache.oozie.command.coord.CoordSubmitXCommand.resolveTagContents(CoordSubmitXCommand.java:819)
... 33 more
Caused by: java.text.ParseException: Unparseable date: "coord:nominalTime()"
at java.text.DateFormat.parse(DateFormat.java:337)
at com.turner.util.Utils.formatTimeTZ(Unknown Source)
at com.turner.util.Utils.formatTimeTZ(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.commons.el.FunctionInvocation.evaluate(FunctionInvocation.java:172)
at org.apache.commons.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:263)
at org.apache.commons.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:190)
at org.apache.oozie.util.ELEvaluator.evaluate(ELEvaluator.java:203)
at org.apache.oozie.coord.CoordELFunctions.evalAndWrap(CoordELFunctions.java:542)
... 34 more
________________________________
From: Mohammad Islam <mi...@yahoo.com>
To: "user@oozie.apache.org" <us...@oozie.apache.org>; Eduardo Afonso Ferreira <ea...@yahoo.com>
Sent: Thursday, November 15, 2012 4:45 PM
Subject: Re: Creating/implementing custom EL Functions.
Hi eduardo,
You have to add the similar thing in three other properties. Please search for "formatTime" in ooze-default.xml. Coordinator EL functions are evaluated in different contexts (multiple phases). Other cases, it called "*echo" method such as "coord_formatTime_echo". For example, during job submission value nominalTime is not defined.
<name>oozie.service.ELService.ext.functions.coord-action-create</name>
and
oozie.service.ELService.ext.functions.coord-action-create-inst
oozie.service.ELService.ext.functions.coord-action-start
Regards,
Mohammad
________________________________
From: Eduardo Afonso Ferreira <ea...@yahoo.com>
To: "user@oozie.apache.org" <us...@oozie.apache.org>
Sent: Thursday, November 15, 2012 1:10 PM
Subject: Re: Creating/implementing custom EL Functions.
Hey,
I'm trying to do some date manipulation via custom EL functions.
I added a function coord:formatTimeTZ() very similar to the original coord:formatTime() but the new one will take an offset and optionally a timezone.
Here's an example of usage (below).
The problem I see now is when I use only the original coord functions, all functions are called and processed fine, but when I use my new function (formatTimeTZ), the attribute coord:nominalTime() is not called, but it simply passes the string "coord:nominalTime()" to the function.
Did I miss anything?
<property>
<name>START_ROW_DT</name>
<value>${coord:dateOffset(coord:nominalTime(), -FREQUENCY-OFFSET, 'MINUTE')}</value>
</property>
<property>
<name>START_ROW_FORM_1</name>
<value>${coord:formatTime(coord:dateOffset(coord:nominalTime(), -FREQUENCY-OFFSET, 'MINUTE'), "'min-'yyyyMMddHHmm")}</value>
</property>
<property>
<name>START_ROW_FORM_2</name>
<value>${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}</value>
</property>
NOTE: Here's what I added to oozie-site.xml to define the new function. I also added my new jar to oozie.war with oozie setup.
<property>
<name>oozie.service.ELService.ext.functions.coord-job-submit-data</name>
<value>
coord:formatTimeTZ=com.turner.util.Utils#formatTimeTZ
</value>
<description>
EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD.
</description>
</property>
________________________________
From: Harsh J <ha...@cloudera.com>
To: user@oozie.apache.org; Eduardo Afonso Ferreira <ea...@yahoo.com>
Sent: Wednesday, November 14, 2012 3:10 PM
Subject: Re: Creating/implementing custom EL Functions.
Hey Eduardo,
There is lack of docs in this area but you can certainly plug in
custom EL functions and give them aliases.
Look at all the configs prefixed
"oozie.service.ELService.ext.functions" in
http://oozie.apache.org/docs/3.2.0-incubating/oozie-default.xml. You
can create your own function that way by mapping a name to a class
plus method call. The class needs to be in the Oozie deployed WAR
though, so this isn't a dynamic thing. See the examples around these
config entries in the link for the inbuilt definitions we carry.
I'd also strongly recommend contributing back nifty EL extensions, so
that its not a maintenance pain in the long run for you guys - and
others can use it as well :)
On Thu, Nov 15, 2012 at 12:58 AM, Eduardo Afonso Ferreira
<ea...@yahoo.com> wrote:
> Hey,
>
> Is it possible to implement custom EL Functions without having to change Oozie source code?
> I did not find documentation about how to implement custom EL Functions.
> Would someone throw some links my way about where I can find information on this?
>
> Thank you.
> Eduardo.
--
Harsh J
Re: Creating/implementing custom EL Functions.
Posted by Eduardo Afonso Ferreira <ea...@yahoo.com>.
Ah, I got it now. And got it to work.
Thank you :)
________________________________
From: Mohammad Islam <mi...@yahoo.com>
To: "user@oozie.apache.org" <us...@oozie.apache.org>; Eduardo Afonso Ferreira <ea...@yahoo.com>
Sent: Thursday, November 15, 2012 6:31 PM
Subject: Re: Creating/implementing custom EL Functions.
Can you please check the "echo" method that I mentioned?
Please check the code at:
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java
and search for "ph1_coord_formatTime_echo". Also check how it is defined in ooze-default.xml.
That means you need to follow the same pattern that we used for formatTime(...)
________________________________
From: Eduardo Afonso Ferreira <ea...@yahoo.com>
To: "user@oozie.apache.org" <us...@oozie.apache.org>
Sent: Thursday, November 15, 2012 2:46 PM
Subject: Re: Creating/implementing custom EL Functions.
Hey,
I added those three but the result appears to be the same.
Here's the stack trace I see in oozie.log. It looks like it does not resolve coord:nominalTime() but just passes that as a string to the function call.
Notice the "Caused by" towards the end of the stack trace:
.....
Caused by: java.text.ParseException: Unparseable date: "coord:nominalTime()"
.....
Here's the stack trace:
2012-11-15 17:40:12,213 WARN V1JobsServlet:542 - USER[aspen] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] URL[POST http://localhost:11000/oozie/v1/jobs] error[E1004], E1004: Expression language evaluation error [U
nable to evaluate :${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}:
], java.lang.Exception: Unable to evaluate :${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}:
org.apache.oozie.servlet.XServletException: E1004: Expression language evaluation error [Unable to evaluate :${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}:
], java.lang.Exception: Unable to evaluate :${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}:
at org.apache.oozie.servlet.V1JobsServlet.submitCoordinatorJob(V1JobsServlet.java:205)
at org.apache.oozie.servlet.V1JobsServlet.submitJob(V1JobsServlet.java:80)
at org.apache.oozie.servlet.BaseJobsServlet.doPost(BaseJobsServlet.java:95)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
at org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:285)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.oozie.servlet.AuthFilter$2.doFilter(AuthFilter.java:126)
at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:372)
at org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:131)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.oozie.servlet.HostnameFilter.doFilter(HostnameFilter.java:67)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.oozie.CoordinatorEngineException: E1004: Expression language evaluation error [Unable to evaluate :${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}:
], java.lang.Exception: Unable to evaluate :${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}:
at org.apache.oozie.CoordinatorEngine.submitJob(CoordinatorEngine.java:374)
at org.apache.oozie.servlet.V1JobsServlet.submitCoordinatorJob(V1JobsServlet.java:200)
... 25 more
Caused by: org.apache.oozie.command.CommandException: E1004: Expression language evaluation error [Unable to evaluate :${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}:
], java.lang.Exception: Unable to evaluate :${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}:
at org.apache.oozie.command.coord.CoordSubmitXCommand.submit(CoordSubmitXCommand.java:272)
at org.apache.oozie.command.SubmitTransitionXCommand.execute(SubmitTransitionXCommand.java:81)
at org.apache.oozie.command.SubmitTransitionXCommand.execute(SubmitTransitionXCommand.java:29)
at org.apache.oozie.command.XCommand.call(XCommand.java:277)
at org.apache.oozie.CoordinatorEngine.submitJob(CoordinatorEngine.java:371)
... 26 more
Caused by: org.apache.oozie.coord.CoordinatorJobException: E1004: Expression language evaluation error [Unable to evaluate :${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}:
], java.lang.Exception: Unable to evaluate :${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}:
at org.apache.oozie.command.coord.CoordSubmitXCommand.resolveTagContents(CoordSubmitXCommand.java:823)
at org.apache.oozie.command.coord.CoordSubmitXCommand.resolveInitial(CoordSubmitXCommand.java:663)
at org.apache.oozie.command.coord.CoordSubmitXCommand.basicResolveAndIncludeDS(CoordSubmitXCommand.java:484)
at org.apache.oozie.command.coord.CoordSubmitXCommand.submit(CoordSubmitXCommand.java:222)
... 30 more
Caused by: java.lang.Exception: Unable to evaluate :${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}:
at org.apache.oozie.coord.CoordELFunctions.evalAndWrap(CoordELFunctions.java:551)
at org.apache.oozie.command.coord.CoordSubmitXCommand.resolveTagContents(CoordSubmitXCommand.java:819)
... 33 more
Caused by: java.text.ParseException: Unparseable date: "coord:nominalTime()"
at java.text.DateFormat.parse(DateFormat.java:337)
at com.turner.util.Utils.formatTimeTZ(Unknown Source)
at com.turner.util.Utils.formatTimeTZ(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.commons.el.FunctionInvocation.evaluate(FunctionInvocation.java:172)
at org.apache.commons.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:263)
at org.apache.commons.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:190)
at org.apache.oozie.util.ELEvaluator.evaluate(ELEvaluator.java:203)
at org.apache.oozie.coord.CoordELFunctions.evalAndWrap(CoordELFunctions.java:542)
... 34 more
________________________________
From: Mohammad Islam <mi...@yahoo.com>
To: "user@oozie.apache.org" <us...@oozie.apache.org>; Eduardo Afonso Ferreira <ea...@yahoo.com>
Sent: Thursday, November 15, 2012 4:45 PM
Subject: Re: Creating/implementing custom EL Functions.
Hi eduardo,
You have to add the similar thing in three other properties. Please search for "formatTime" in ooze-default.xml. Coordinator EL functions are evaluated in different contexts (multiple phases). Other cases, it called "*echo" method such as "coord_formatTime_echo". For example, during job submission value nominalTime is not defined.
<name>oozie.service.ELService.ext.functions.coord-action-create</name>
and
oozie.service.ELService.ext.functions.coord-action-create-inst
oozie.service.ELService.ext.functions.coord-action-start
Regards,
Mohammad
________________________________
From: Eduardo Afonso Ferreira <ea...@yahoo.com>
To: "user@oozie.apache.org" <us...@oozie.apache.org>
Sent: Thursday, November 15, 2012 1:10 PM
Subject: Re: Creating/implementing custom EL Functions.
Hey,
I'm trying to do some date manipulation via custom EL functions.
I added a function coord:formatTimeTZ() very similar to the original coord:formatTime() but the new one will take an offset and optionally a timezone.
Here's an example of usage (below).
The problem I see now is when I use only the original coord functions, all functions are called and processed fine, but when I use my new function (formatTimeTZ), the attribute coord:nominalTime() is not called, but it simply passes the string "coord:nominalTime()" to the function.
Did I miss anything?
<property>
<name>START_ROW_DT</name>
<value>${coord:dateOffset(coord:nominalTime(), -FREQUENCY-OFFSET, 'MINUTE')}</value>
</property>
<property>
<name>START_ROW_FORM_1</name>
<value>${coord:formatTime(coord:dateOffset(coord:nominalTime(), -FREQUENCY-OFFSET, 'MINUTE'), "'min-'yyyyMMddHHmm")}</value>
</property>
<property>
<name>START_ROW_FORM_2</name>
<value>${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}</value>
</property>
NOTE: Here's what I added to oozie-site.xml to define the new function. I also added my new jar to oozie.war with oozie setup.
<property>
<name>oozie.service.ELService.ext.functions.coord-job-submit-data</name>
<value>
coord:formatTimeTZ=com.turner.util.Utils#formatTimeTZ
</value>
<description>
EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD.
</description>
</property>
________________________________
From: Harsh J <ha...@cloudera.com>
To: user@oozie.apache.org; Eduardo Afonso Ferreira <ea...@yahoo.com>
Sent: Wednesday, November 14, 2012 3:10 PM
Subject: Re: Creating/implementing custom EL Functions.
Hey Eduardo,
There is lack of docs in this area but you can certainly plug in
custom EL functions and give them aliases.
Look at all the configs prefixed
"oozie.service.ELService.ext.functions" in
http://oozie.apache.org/docs/3.2.0-incubating/oozie-default.xml. You
can create your own function that way by mapping a name to a class
plus method call. The class needs to be in the Oozie deployed WAR
though, so this isn't a dynamic thing. See the examples around these
config entries in the link for the inbuilt definitions we carry.
I'd also strongly recommend contributing back nifty EL extensions, so
that its not a maintenance pain in the long run for you guys - and
others can use it as well :)
On Thu, Nov 15, 2012 at 12:58 AM, Eduardo Afonso Ferreira
<ea...@yahoo.com> wrote:
> Hey,
>
> Is it possible to implement custom EL Functions without having to change Oozie source code?
> I did not find documentation about how to implement custom EL Functions.
> Would someone throw some links my way about where I can find information on this?
>
> Thank you.
> Eduardo.
--
Harsh J
Re: Creating/implementing custom EL Functions.
Posted by Mohammad Islam <mi...@yahoo.com>.
Can you please check the "echo" method that I mentioned?
Please check the code at:
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java
and search for "ph1_coord_formatTime_echo". Also check how it is defined in ooze-default.xml.
That means you need to follow the same pattern that we used for formatTime(...)
________________________________
From: Eduardo Afonso Ferreira <ea...@yahoo.com>
To: "user@oozie.apache.org" <us...@oozie.apache.org>
Sent: Thursday, November 15, 2012 2:46 PM
Subject: Re: Creating/implementing custom EL Functions.
Hey,
I added those three but the result appears to be the same.
Here's the stack trace I see in oozie.log. It looks like it does not resolve coord:nominalTime() but just passes that as a string to the function call.
Notice the "Caused by" towards the end of the stack trace:
.....
Caused by: java.text.ParseException: Unparseable date: "coord:nominalTime()"
.....
Here's the stack trace:
2012-11-15 17:40:12,213 WARN V1JobsServlet:542 - USER[aspen] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] URL[POST http://localhost:11000/oozie/v1/jobs] error[E1004], E1004: Expression language evaluation error [U
nable to evaluate :${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}:
], java.lang.Exception: Unable to evaluate :${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}:
org.apache.oozie.servlet.XServletException: E1004: Expression language evaluation error [Unable to evaluate :${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}:
], java.lang.Exception: Unable to evaluate :${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}:
at org.apache.oozie.servlet.V1JobsServlet.submitCoordinatorJob(V1JobsServlet.java:205)
at org.apache.oozie.servlet.V1JobsServlet.submitJob(V1JobsServlet.java:80)
at org.apache.oozie.servlet.BaseJobsServlet.doPost(BaseJobsServlet.java:95)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
at org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:285)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.oozie.servlet.AuthFilter$2.doFilter(AuthFilter.java:126)
at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:372)
at org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:131)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.oozie.servlet.HostnameFilter.doFilter(HostnameFilter.java:67)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.oozie.CoordinatorEngineException: E1004: Expression language evaluation error [Unable to evaluate :${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}:
], java.lang.Exception: Unable to evaluate :${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}:
at org.apache.oozie.CoordinatorEngine.submitJob(CoordinatorEngine.java:374)
at org.apache.oozie.servlet.V1JobsServlet.submitCoordinatorJob(V1JobsServlet.java:200)
... 25 more
Caused by: org.apache.oozie.command.CommandException: E1004: Expression language evaluation error [Unable to evaluate :${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}:
], java.lang.Exception: Unable to evaluate :${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}:
at org.apache.oozie.command.coord.CoordSubmitXCommand.submit(CoordSubmitXCommand.java:272)
at org.apache.oozie.command.SubmitTransitionXCommand.execute(SubmitTransitionXCommand.java:81)
at org.apache.oozie.command.SubmitTransitionXCommand.execute(SubmitTransitionXCommand.java:29)
at org.apache.oozie.command.XCommand.call(XCommand.java:277)
at org.apache.oozie.CoordinatorEngine.submitJob(CoordinatorEngine.java:371)
... 26 more
Caused by: org.apache.oozie.coord.CoordinatorJobException: E1004: Expression language evaluation error [Unable to evaluate :${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}:
], java.lang.Exception: Unable to evaluate :${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}:
at org.apache.oozie.command.coord.CoordSubmitXCommand.resolveTagContents(CoordSubmitXCommand.java:823)
at org.apache.oozie.command.coord.CoordSubmitXCommand.resolveInitial(CoordSubmitXCommand.java:663)
at org.apache.oozie.command.coord.CoordSubmitXCommand.basicResolveAndIncludeDS(CoordSubmitXCommand.java:484)
at org.apache.oozie.command.coord.CoordSubmitXCommand.submit(CoordSubmitXCommand.java:222)
... 30 more
Caused by: java.lang.Exception: Unable to evaluate :${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}:
at org.apache.oozie.coord.CoordELFunctions.evalAndWrap(CoordELFunctions.java:551)
at org.apache.oozie.command.coord.CoordSubmitXCommand.resolveTagContents(CoordSubmitXCommand.java:819)
... 33 more
Caused by: java.text.ParseException: Unparseable date: "coord:nominalTime()"
at java.text.DateFormat.parse(DateFormat.java:337)
at com.turner.util.Utils.formatTimeTZ(Unknown Source)
at com.turner.util.Utils.formatTimeTZ(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.commons.el.FunctionInvocation.evaluate(FunctionInvocation.java:172)
at org.apache.commons.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:263)
at org.apache.commons.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:190)
at org.apache.oozie.util.ELEvaluator.evaluate(ELEvaluator.java:203)
at org.apache.oozie.coord.CoordELFunctions.evalAndWrap(CoordELFunctions.java:542)
... 34 more
________________________________
From: Mohammad Islam <mi...@yahoo.com>
To: "user@oozie.apache.org" <us...@oozie.apache.org>; Eduardo Afonso Ferreira <ea...@yahoo.com>
Sent: Thursday, November 15, 2012 4:45 PM
Subject: Re: Creating/implementing custom EL Functions.
Hi eduardo,
You have to add the similar thing in three other properties. Please search for "formatTime" in ooze-default.xml. Coordinator EL functions are evaluated in different contexts (multiple phases). Other cases, it called "*echo" method such as "coord_formatTime_echo". For example, during job submission value nominalTime is not defined.
<name>oozie.service.ELService.ext.functions.coord-action-create</name>
and
oozie.service.ELService.ext.functions.coord-action-create-inst
oozie.service.ELService.ext.functions.coord-action-start
Regards,
Mohammad
________________________________
From: Eduardo Afonso Ferreira <ea...@yahoo.com>
To: "user@oozie.apache.org" <us...@oozie.apache.org>
Sent: Thursday, November 15, 2012 1:10 PM
Subject: Re: Creating/implementing custom EL Functions.
Hey,
I'm trying to do some date manipulation via custom EL functions.
I added a function coord:formatTimeTZ() very similar to the original coord:formatTime() but the new one will take an offset and optionally a timezone.
Here's an example of usage (below).
The problem I see now is when I use only the original coord functions, all functions are called and processed fine, but when I use my new function (formatTimeTZ), the attribute coord:nominalTime() is not called, but it simply passes the string "coord:nominalTime()" to the function.
Did I miss anything?
<property>
<name>START_ROW_DT</name>
<value>${coord:dateOffset(coord:nominalTime(), -FREQUENCY-OFFSET, 'MINUTE')}</value>
</property>
<property>
<name>START_ROW_FORM_1</name>
<value>${coord:formatTime(coord:dateOffset(coord:nominalTime(), -FREQUENCY-OFFSET, 'MINUTE'), "'min-'yyyyMMddHHmm")}</value>
</property>
<property>
<name>START_ROW_FORM_2</name>
<value>${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}</value>
</property>
NOTE: Here's what I added to oozie-site.xml to define the new function. I also added my new jar to oozie.war with oozie setup.
<property>
<name>oozie.service.ELService.ext.functions.coord-job-submit-data</name>
<value>
coord:formatTimeTZ=com.turner.util.Utils#formatTimeTZ
</value>
<description>
EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD.
</description>
</property>
________________________________
From: Harsh J <ha...@cloudera.com>
To: user@oozie.apache.org; Eduardo Afonso Ferreira <ea...@yahoo.com>
Sent: Wednesday, November 14, 2012 3:10 PM
Subject: Re: Creating/implementing custom EL Functions.
Hey Eduardo,
There is lack of docs in this area but you can certainly plug in
custom EL functions and give them aliases.
Look at all the configs prefixed
"oozie.service.ELService.ext.functions" in
http://oozie.apache.org/docs/3.2.0-incubating/oozie-default.xml. You
can create your own function that way by mapping a name to a class
plus method call. The class needs to be in the Oozie deployed WAR
though, so this isn't a dynamic thing. See the examples around these
config entries in the link for the inbuilt definitions we carry.
I'd also strongly recommend contributing back nifty EL extensions, so
that its not a maintenance pain in the long run for you guys - and
others can use it as well :)
On Thu, Nov 15, 2012 at 12:58 AM, Eduardo Afonso Ferreira
<ea...@yahoo.com> wrote:
> Hey,
>
> Is it possible to implement custom EL Functions without having to change Oozie source code?
> I did not find documentation about how to implement custom EL Functions.
> Would someone throw some links my way about where I can find information on this?
>
> Thank you.
> Eduardo.
--
Harsh J
Re: Creating/implementing custom EL Functions.
Posted by Eduardo Afonso Ferreira <ea...@yahoo.com>.
Hey,
I added those three but the result appears to be the same.
Here's the stack trace I see in oozie.log. It looks like it does not resolve coord:nominalTime() but just passes that as a string to the function call.
Notice the "Caused by" towards the end of the stack trace:
.....
Caused by: java.text.ParseException: Unparseable date: "coord:nominalTime()"
.....
Here's the stack trace:
2012-11-15 17:40:12,213 WARN V1JobsServlet:542 - USER[aspen] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] URL[POST http://localhost:11000/oozie/v1/jobs] error[E1004], E1004: Expression language evaluation error [U
nable to evaluate :${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}:
], java.lang.Exception: Unable to evaluate :${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}:
org.apache.oozie.servlet.XServletException: E1004: Expression language evaluation error [Unable to evaluate :${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}:
], java.lang.Exception: Unable to evaluate :${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}:
at org.apache.oozie.servlet.V1JobsServlet.submitCoordinatorJob(V1JobsServlet.java:205)
at org.apache.oozie.servlet.V1JobsServlet.submitJob(V1JobsServlet.java:80)
at org.apache.oozie.servlet.BaseJobsServlet.doPost(BaseJobsServlet.java:95)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
at org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:285)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.oozie.servlet.AuthFilter$2.doFilter(AuthFilter.java:126)
at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:372)
at org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:131)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.oozie.servlet.HostnameFilter.doFilter(HostnameFilter.java:67)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.oozie.CoordinatorEngineException: E1004: Expression language evaluation error [Unable to evaluate :${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}:
], java.lang.Exception: Unable to evaluate :${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}:
at org.apache.oozie.CoordinatorEngine.submitJob(CoordinatorEngine.java:374)
at org.apache.oozie.servlet.V1JobsServlet.submitCoordinatorJob(V1JobsServlet.java:200)
... 25 more
Caused by: org.apache.oozie.command.CommandException: E1004: Expression language evaluation error [Unable to evaluate :${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}:
], java.lang.Exception: Unable to evaluate :${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}:
at org.apache.oozie.command.coord.CoordSubmitXCommand.submit(CoordSubmitXCommand.java:272)
at org.apache.oozie.command.SubmitTransitionXCommand.execute(SubmitTransitionXCommand.java:81)
at org.apache.oozie.command.SubmitTransitionXCommand.execute(SubmitTransitionXCommand.java:29)
at org.apache.oozie.command.XCommand.call(XCommand.java:277)
at org.apache.oozie.CoordinatorEngine.submitJob(CoordinatorEngine.java:371)
... 26 more
Caused by: org.apache.oozie.coord.CoordinatorJobException: E1004: Expression language evaluation error [Unable to evaluate :${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}:
], java.lang.Exception: Unable to evaluate :${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}:
at org.apache.oozie.command.coord.CoordSubmitXCommand.resolveTagContents(CoordSubmitXCommand.java:823)
at org.apache.oozie.command.coord.CoordSubmitXCommand.resolveInitial(CoordSubmitXCommand.java:663)
at org.apache.oozie.command.coord.CoordSubmitXCommand.basicResolveAndIncludeDS(CoordSubmitXCommand.java:484)
at org.apache.oozie.command.coord.CoordSubmitXCommand.submit(CoordSubmitXCommand.java:222)
... 30 more
Caused by: java.lang.Exception: Unable to evaluate :${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}:
at org.apache.oozie.coord.CoordELFunctions.evalAndWrap(CoordELFunctions.java:551)
at org.apache.oozie.command.coord.CoordSubmitXCommand.resolveTagContents(CoordSubmitXCommand.java:819)
... 33 more
Caused by: java.text.ParseException: Unparseable date: "coord:nominalTime()"
at java.text.DateFormat.parse(DateFormat.java:337)
at com.turner.util.Utils.formatTimeTZ(Unknown Source)
at com.turner.util.Utils.formatTimeTZ(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.commons.el.FunctionInvocation.evaluate(FunctionInvocation.java:172)
at org.apache.commons.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:263)
at org.apache.commons.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:190)
at org.apache.oozie.util.ELEvaluator.evaluate(ELEvaluator.java:203)
at org.apache.oozie.coord.CoordELFunctions.evalAndWrap(CoordELFunctions.java:542)
... 34 more
________________________________
From: Mohammad Islam <mi...@yahoo.com>
To: "user@oozie.apache.org" <us...@oozie.apache.org>; Eduardo Afonso Ferreira <ea...@yahoo.com>
Sent: Thursday, November 15, 2012 4:45 PM
Subject: Re: Creating/implementing custom EL Functions.
Hi eduardo,
You have to add the similar thing in three other properties. Please search for "formatTime" in ooze-default.xml. Coordinator EL functions are evaluated in different contexts (multiple phases). Other cases, it called "*echo" method such as "coord_formatTime_echo". For example, during job submission value nominalTime is not defined.
<name>oozie.service.ELService.ext.functions.coord-action-create</name>
and
oozie.service.ELService.ext.functions.coord-action-create-inst
oozie.service.ELService.ext.functions.coord-action-start
Regards,
Mohammad
________________________________
From: Eduardo Afonso Ferreira <ea...@yahoo.com>
To: "user@oozie.apache.org" <us...@oozie.apache.org>
Sent: Thursday, November 15, 2012 1:10 PM
Subject: Re: Creating/implementing custom EL Functions.
Hey,
I'm trying to do some date manipulation via custom EL functions.
I added a function coord:formatTimeTZ() very similar to the original coord:formatTime() but the new one will take an offset and optionally a timezone.
Here's an example of usage (below).
The problem I see now is when I use only the original coord functions, all functions are called and processed fine, but when I use my new function (formatTimeTZ), the attribute coord:nominalTime() is not called, but it simply passes the string "coord:nominalTime()" to the function.
Did I miss anything?
<property>
<name>START_ROW_DT</name>
<value>${coord:dateOffset(coord:nominalTime(), -FREQUENCY-OFFSET, 'MINUTE')}</value>
</property>
<property>
<name>START_ROW_FORM_1</name>
<value>${coord:formatTime(coord:dateOffset(coord:nominalTime(), -FREQUENCY-OFFSET, 'MINUTE'), "'min-'yyyyMMddHHmm")}</value>
</property>
<property>
<name>START_ROW_FORM_2</name>
<value>${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}</value>
</property>
NOTE: Here's what I added to oozie-site.xml to define the new function. I also added my new jar to oozie.war with oozie setup.
<property>
<name>oozie.service.ELService.ext.functions.coord-job-submit-data</name>
<value>
coord:formatTimeTZ=com.turner.util.Utils#formatTimeTZ
</value>
<description>
EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD.
</description>
</property>
________________________________
From: Harsh J <ha...@cloudera.com>
To: user@oozie.apache.org; Eduardo Afonso Ferreira <ea...@yahoo.com>
Sent: Wednesday, November 14, 2012 3:10 PM
Subject: Re: Creating/implementing custom EL Functions.
Hey Eduardo,
There is lack of docs in this area but you can certainly plug in
custom EL functions and give them aliases.
Look at all the configs prefixed
"oozie.service.ELService.ext.functions" in
http://oozie.apache.org/docs/3.2.0-incubating/oozie-default.xml. You
can create your own function that way by mapping a name to a class
plus method call. The class needs to be in the Oozie deployed WAR
though, so this isn't a dynamic thing. See the examples around these
config entries in the link for the inbuilt definitions we carry.
I'd also strongly recommend contributing back nifty EL extensions, so
that its not a maintenance pain in the long run for you guys - and
others can use it as well :)
On Thu, Nov 15, 2012 at 12:58 AM, Eduardo Afonso Ferreira
<ea...@yahoo.com> wrote:
> Hey,
>
> Is it possible to implement custom EL Functions without having to change Oozie source code?
> I did not find documentation about how to implement custom EL Functions.
> Would someone throw some links my way about where I can find information on this?
>
> Thank you.
> Eduardo.
--
Harsh J
Re: Creating/implementing custom EL Functions.
Posted by Mohammad Islam <mi...@yahoo.com>.
Hi eduardo,
You have to add the similar thing in three other properties. Please search for "formatTime" in ooze-default.xml. Coordinator EL functions are evaluated in different contexts (multiple phases). Other cases, it called "*echo" method such as "coord_formatTime_echo". For example, during job submission value nominalTime is not defined.
<name>oozie.service.ELService.ext.functions.coord-action-create</name>
and
oozie.service.ELService.ext.functions.coord-action-create-inst
oozie.service.ELService.ext.functions.coord-action-start
Regards,
Mohammad
________________________________
From: Eduardo Afonso Ferreira <ea...@yahoo.com>
To: "user@oozie.apache.org" <us...@oozie.apache.org>
Sent: Thursday, November 15, 2012 1:10 PM
Subject: Re: Creating/implementing custom EL Functions.
Hey,
I'm trying to do some date manipulation via custom EL functions.
I added a function coord:formatTimeTZ() very similar to the original coord:formatTime() but the new one will take an offset and optionally a timezone.
Here's an example of usage (below).
The problem I see now is when I use only the original coord functions, all functions are called and processed fine, but when I use my new function (formatTimeTZ), the attribute coord:nominalTime() is not called, but it simply passes the string "coord:nominalTime()" to the function.
Did I miss anything?
<property>
<name>START_ROW_DT</name>
<value>${coord:dateOffset(coord:nominalTime(), -FREQUENCY-OFFSET, 'MINUTE')}</value>
</property>
<property>
<name>START_ROW_FORM_1</name>
<value>${coord:formatTime(coord:dateOffset(coord:nominalTime(), -FREQUENCY-OFFSET, 'MINUTE'), "'min-'yyyyMMddHHmm")}</value>
</property>
<property>
<name>START_ROW_FORM_2</name>
<value>${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}</value>
</property>
NOTE: Here's what I added to oozie-site.xml to define the new function. I also added my new jar to oozie.war with oozie setup.
<property>
<name>oozie.service.ELService.ext.functions.coord-job-submit-data</name>
<value>
coord:formatTimeTZ=com.turner.util.Utils#formatTimeTZ
</value>
<description>
EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD.
</description>
</property>
________________________________
From: Harsh J <ha...@cloudera.com>
To: user@oozie.apache.org; Eduardo Afonso Ferreira <ea...@yahoo.com>
Sent: Wednesday, November 14, 2012 3:10 PM
Subject: Re: Creating/implementing custom EL Functions.
Hey Eduardo,
There is lack of docs in this area but you can certainly plug in
custom EL functions and give them aliases.
Look at all the configs prefixed
"oozie.service.ELService.ext.functions" in
http://oozie.apache.org/docs/3.2.0-incubating/oozie-default.xml. You
can create your own function that way by mapping a name to a class
plus method call. The class needs to be in the Oozie deployed WAR
though, so this isn't a dynamic thing. See the examples around these
config entries in the link for the inbuilt definitions we carry.
I'd also strongly recommend contributing back nifty EL extensions, so
that its not a maintenance pain in the long run for you guys - and
others can use it as well :)
On Thu, Nov 15, 2012 at 12:58 AM, Eduardo Afonso Ferreira
<ea...@yahoo.com> wrote:
> Hey,
>
> Is it possible to implement custom EL Functions without having to change Oozie source code?
> I did not find documentation about how to implement custom EL Functions.
> Would someone throw some links my way about where I can find information on this?
>
> Thank you.
> Eduardo.
--
Harsh J
Re: Creating/implementing custom EL Functions.
Posted by Eduardo Afonso Ferreira <ea...@yahoo.com>.
Hey,
I'm trying to do some date manipulation via custom EL functions.
I added a function coord:formatTimeTZ() very similar to the original coord:formatTime() but the new one will take an offset and optionally a timezone.
Here's an example of usage (below).
The problem I see now is when I use only the original coord functions, all functions are called and processed fine, but when I use my new function (formatTimeTZ), the attribute coord:nominalTime() is not called, but it simply passes the string "coord:nominalTime()" to the function.
Did I miss anything?
<property>
<name>START_ROW_DT</name>
<value>${coord:dateOffset(coord:nominalTime(), -FREQUENCY-OFFSET, 'MINUTE')}</value>
</property>
<property>
<name>START_ROW_FORM_1</name>
<value>${coord:formatTime(coord:dateOffset(coord:nominalTime(), -FREQUENCY-OFFSET, 'MINUTE'), "'min-'yyyyMMddHHmm")}</value>
</property>
<property>
<name>START_ROW_FORM_2</name>
<value>${coord:formatTimeTZ(coord:nominalTime(), -FREQUENCY-OFFSET, "'min-'yyyyMMddHHmm")}</value>
</property>
NOTE: Here's what I added to oozie-site.xml to define the new function. I also added my new jar to oozie.war with oozie setup.
<property>
<name>oozie.service.ELService.ext.functions.coord-job-submit-data</name>
<value>
coord:formatTimeTZ=com.turner.util.Utils#formatTimeTZ
</value>
<description>
EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD.
</description>
</property>
________________________________
From: Harsh J <ha...@cloudera.com>
To: user@oozie.apache.org; Eduardo Afonso Ferreira <ea...@yahoo.com>
Sent: Wednesday, November 14, 2012 3:10 PM
Subject: Re: Creating/implementing custom EL Functions.
Hey Eduardo,
There is lack of docs in this area but you can certainly plug in
custom EL functions and give them aliases.
Look at all the configs prefixed
"oozie.service.ELService.ext.functions" in
http://oozie.apache.org/docs/3.2.0-incubating/oozie-default.xml. You
can create your own function that way by mapping a name to a class
plus method call. The class needs to be in the Oozie deployed WAR
though, so this isn't a dynamic thing. See the examples around these
config entries in the link for the inbuilt definitions we carry.
I'd also strongly recommend contributing back nifty EL extensions, so
that its not a maintenance pain in the long run for you guys - and
others can use it as well :)
On Thu, Nov 15, 2012 at 12:58 AM, Eduardo Afonso Ferreira
<ea...@yahoo.com> wrote:
> Hey,
>
> Is it possible to implement custom EL Functions without having to change Oozie source code?
> I did not find documentation about how to implement custom EL Functions.
> Would someone throw some links my way about where I can find information on this?
>
> Thank you.
> Eduardo.
--
Harsh J
Re: Creating/implementing custom EL Functions.
Posted by Harsh J <ha...@cloudera.com>.
Hey Eduardo,
There is lack of docs in this area but you can certainly plug in
custom EL functions and give them aliases.
Look at all the configs prefixed
"oozie.service.ELService.ext.functions" in
http://oozie.apache.org/docs/3.2.0-incubating/oozie-default.xml. You
can create your own function that way by mapping a name to a class
plus method call. The class needs to be in the Oozie deployed WAR
though, so this isn't a dynamic thing. See the examples around these
config entries in the link for the inbuilt definitions we carry.
I'd also strongly recommend contributing back nifty EL extensions, so
that its not a maintenance pain in the long run for you guys - and
others can use it as well :)
On Thu, Nov 15, 2012 at 12:58 AM, Eduardo Afonso Ferreira
<ea...@yahoo.com> wrote:
> Hey,
>
> Is it possible to implement custom EL Functions without having to change Oozie source code?
> I did not find documentation about how to implement custom EL Functions.
> Would someone throw some links my way about where I can find information on this?
>
> Thank you.
> Eduardo.
--
Harsh J