You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by Sunil B <bs...@gmail.com> on 2015/10/19 16:27:22 UTC
parameter/variable substitution not working in oozie 4.0.0-cdh5.3.2
Hi,
My Oozie server version: 4.0.0-cdh5.3.2
I have a variable A, the value of which would be a combination of
other variables (B, C etc.). However, oozie is unable to recognize and
substitute values from B, C to form value of A. It throws:
javax.servlet.jsp.el.ELException: variable [param1] cannot be resolved
Please let me know if it is a known issue in the version I am using.
Here is a simple workflow example I am trying:
<workflow-app name="test-sunil-wf" xmlns="uri:oozie:workflow:0.4">
<global>
<configuration>
<property>
<name>param1</name>
<value>idHere</value>
</property>
</configuration>
</global>
<start to="test-echo-shell-1"/>
<action name="test-echo-shell-1">
<shell xmlns="uri:oozie:shell-action:0.1">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<exec>/bin/echo</exec>
<argument>${param1}</argument>
<capture-output/>
</shell>
<ok to="end"/>
<error to="kill"/>
</action>
<kill name="kill">
<message>Action failed, error
message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<end name="end"/>
</workflow-app>
The above example throws:
javax.servlet.jsp.el.ELException: variable [param1] cannot be resolved
at org.apache.oozie.util.ELEvaluator$Context.resolveVariable(ELEvaluator.java:106)
at org.apache.commons.el.NamedValue.evaluate(NamedValue.java:124)
at org.apache.commons.el.ExpressionString.evaluate(ExpressionString.java:114)
at org.apache.commons.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:274)
at org.apache.commons.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:190)
at org.apache.oozie.util.ELEvaluator.evaluate(ELEvaluator.java:203)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:188)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:63)
at org.apache.oozie.command.XCommand.call(XCommand.java:281)
at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:323)
at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:252)
at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:174)
Here is another example which fails. This is somewhat similar to my
real use-case:
<workflow-app name="test-sunil-wf" xmlns="uri:oozie:workflow:0.4">
<start to="test-echo-shell-1"/>
<action name="test-echo-shell-1">
<shell xmlns="uri:oozie:shell-action:0.1">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>param1</name>
<value>${wf:id()}</value>
</property>
<property>
<name>param2</name>
<value>${wf:name()}</value>
</property>
<property>
<name>param3</name>
<value>${param1} and ${param2}</value>
</property>
</configuration>
<exec>/bin/echo</exec>
<argument>${param3}</argument>
<capture-output/>
</shell>
<ok to="end"/>
<error to="kill"/>
</action>
<kill name="kill">
<message>Action failed, error
message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<end name="end"/>
</workflow-app>
The above XML fails with the same stacktrace as shown above.
Thanks
Sunil
Re: parameter/variable substitution not working in oozie 4.0.0-cdh5.3.2
Posted by Robert Kanter <rk...@cloudera.com>.
I don't think there's currently a way to do what you want. There is a
Formal Parameters feature [1], where you put a <parameters> section at the
top of your workflow where you can define required properties and default
values for them, but I don't think those default values can be EL
Functions. If you'd like that ability, please file a JIRA with the
details.
As a workaround, you could create a workflow that has the EL functions you
want, and then uses the subworkflow action to pass them to another workflow
where they'll be resolved, so you should be able to get the behavior you
want this way. The only downside is that you have to do it through a
subworkflow.
[1]
http://oozie.apache.org/docs/4.2.0/WorkflowFunctionalSpec.html#a4.1_Workflow_Job_Properties_or_Parameters
On Mon, Oct 19, 2015 at 10:34 AM, Sunil B <bs...@gmail.com> wrote:
> Thanks for your reply Oussama. However, based on what I have read It
> should be possible to access configuration properties in your
> workflow.xml with the help of EL.
> Reference: "Apache Oozie" book from O'Reilly, Example 5-13
> Possible Source code reference: substituteVars function in
>
> https://github.com/apache/oozie/blob/release-4.0.0/core/src/main/java/org/apache/oozie/util/XConfiguration.java
>
> If this feature is not present in Oozie, then the use of
> ${wf:conf("prop.name")} is very limited.
>
> Let me describe my situation as closely as possible:
> I need to copy a source file to 3 or 4 different destinations.
> All the paths follow similar convention like:
> <namenode>/<parentdir>/yyyy/yyyymm/yyyymmdd/<childdir>/<fileName>
> I am using ${coord:formatTime(coord:nominalTime(), "yyyy")} to get the
> yyyy string and similarly for mm and dd part of the path.
> What I would like to do is use the coord:formatTime function as few
> places as possible to make it easy to change in future. For example:
> <configuration>
> <property>
> <name>year</name>
> <value>${coord:formatTime(coord:nominalTime(), "yyyy")}</value>
> </property>
> <property>
> <name>month</name>
> <value>${coord:formatTime(coord:nominalTime(), "MM")}</value>
> </property>
> <property>
> <name>date</name>
> <value>${coord:formatTime(coord:nominalTime(), "dd")}</value>
> </property>
> <property>
> <name>path_containing_date</name>
> <value>${year}/${year}${month}/${year}${month}${date}</value>
> </property>
> <property>
> <name>source_path</name>
>
> <value>${namenodeA_with_parent_path}/${path_containing_date}/${remaining_path}</value>
> </property>
> <property>
> <name>destination_path_1</name>
>
> <value>${namenode1_with_parent_path}/${path_containing_date}/${remaining_path}</value>
> </property>
> <property>
> <name>destination_path_2</name>
>
> <value>${namenode2_with_parent_path}/${path_containing_date}/${remaining_path}</value>
> </property>
> <property>
> <name>destination_path_3</name>
>
> <value>${namenode3_with_parent_path}/${path_containing_date}/${remaining_path}</value>
> </property>
> </configuration>
>
>
> This I hope would be more maintainable than using formatTime
> everywhere I need date. This way If I want to run manually for a
> different date, I can just pass in the year, month and date values.
>
> If this feature is not present right now, Is there a plan to implement it?
>
> On Mon, Oct 19, 2015 at 8:06 AM, Oussama Chougna
> <ou...@hotmail.com> wrote:
> > Hello Sunil,
> > That's not how it works. In the global section you can define oozie
> properties which are global for all actions in the workflow, like
> mapred.job.queue.name, job-xml etc..
> > What you can do is put 'param1' in your job.properties, then you can
> resolve it with EL via ${param1}.
> >
> > Best,
> >
> > Oussama Chougna
> >
> >
> >> Date: Mon, 19 Oct 2015 07:27:22 -0700
> >> Subject: parameter/variable substitution not working in oozie
> 4.0.0-cdh5.3.2
> >> From: bsunil.sp@gmail.com
> >> To: user@oozie.apache.org
> >>
> >> Hi,
> >>
> >> My Oozie server version: 4.0.0-cdh5.3.2
> >> I have a variable A, the value of which would be a combination of
> >> other variables (B, C etc.). However, oozie is unable to recognize and
> >> substitute values from B, C to form value of A. It throws:
> >> javax.servlet.jsp.el.ELException: variable [param1] cannot be resolved
> >>
> >> Please let me know if it is a known issue in the version I am using.
> >>
> >> Here is a simple workflow example I am trying:
> >> <workflow-app name="test-sunil-wf" xmlns="uri:oozie:workflow:0.4">
> >> <global>
> >> <configuration>
> >> <property>
> >> <name>param1</name>
> >> <value>idHere</value>
> >> </property>
> >> </configuration>
> >> </global>
> >> <start to="test-echo-shell-1"/>
> >> <action name="test-echo-shell-1">
> >> <shell xmlns="uri:oozie:shell-action:0.1">
> >> <job-tracker>${jobTracker}</job-tracker>
> >> <name-node>${nameNode}</name-node>
> >> <exec>/bin/echo</exec>
> >> <argument>${param1}</argument>
> >> <capture-output/>
> >> </shell>
> >> <ok to="end"/>
> >> <error to="kill"/>
> >> </action>
> >> <kill name="kill">
> >> <message>Action failed, error
> >> message[${wf:errorMessage(wf:lastErrorNode())}]</message>
> >> </kill>
> >> <end name="end"/>
> >> </workflow-app>
> >>
> >> The above example throws:
> >> javax.servlet.jsp.el.ELException: variable [param1] cannot be resolved
> >> at
> org.apache.oozie.util.ELEvaluator$Context.resolveVariable(ELEvaluator.java:106)
> >> at org.apache.commons.el.NamedValue.evaluate(NamedValue.java:124)
> >> at
> org.apache.commons.el.ExpressionString.evaluate(ExpressionString.java:114)
> >> at
> org.apache.commons.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:274)
> >> at
> org.apache.commons.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:190)
> >> at org.apache.oozie.util.ELEvaluator.evaluate(ELEvaluator.java:203)
> >> at
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:188)
> >> at
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:63)
> >> at org.apache.oozie.command.XCommand.call(XCommand.java:281)
> >> at
> org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:323)
> >> at
> org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:252)
> >> at
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:174)
> >>
> >>
> >> Here is another example which fails. This is somewhat similar to my
> >> real use-case:
> >> <workflow-app name="test-sunil-wf" xmlns="uri:oozie:workflow:0.4">
> >> <start to="test-echo-shell-1"/>
> >> <action name="test-echo-shell-1">
> >> <shell xmlns="uri:oozie:shell-action:0.1">
> >> <job-tracker>${jobTracker}</job-tracker>
> >> <name-node>${nameNode}</name-node>
> >> <configuration>
> >> <property>
> >> <name>param1</name>
> >> <value>${wf:id()}</value>
> >> </property>
> >> <property>
> >> <name>param2</name>
> >> <value>${wf:name()}</value>
> >> </property>
> >> <property>
> >> <name>param3</name>
> >> <value>${param1} and ${param2}</value>
> >> </property>
> >> </configuration>
> >> <exec>/bin/echo</exec>
> >> <argument>${param3}</argument>
> >> <capture-output/>
> >> </shell>
> >> <ok to="end"/>
> >> <error to="kill"/>
> >> </action>
> >> <kill name="kill">
> >> <message>Action failed, error
> >> message[${wf:errorMessage(wf:lastErrorNode())}]</message>
> >> </kill>
> >> <end name="end"/>
> >> </workflow-app>
> >>
> >> The above XML fails with the same stacktrace as shown above.
> >>
> >> Thanks
> >> Sunil
> >
>
Re: parameter/variable substitution not working in oozie 4.0.0-cdh5.3.2
Posted by Sunil B <bs...@gmail.com>.
Thanks for your reply Oussama. However, based on what I have read It
should be possible to access configuration properties in your
workflow.xml with the help of EL.
Reference: "Apache Oozie" book from O'Reilly, Example 5-13
Possible Source code reference: substituteVars function in
https://github.com/apache/oozie/blob/release-4.0.0/core/src/main/java/org/apache/oozie/util/XConfiguration.java
If this feature is not present in Oozie, then the use of
${wf:conf("prop.name")} is very limited.
Let me describe my situation as closely as possible:
I need to copy a source file to 3 or 4 different destinations.
All the paths follow similar convention like:
<namenode>/<parentdir>/yyyy/yyyymm/yyyymmdd/<childdir>/<fileName>
I am using ${coord:formatTime(coord:nominalTime(), "yyyy")} to get the
yyyy string and similarly for mm and dd part of the path.
What I would like to do is use the coord:formatTime function as few
places as possible to make it easy to change in future. For example:
<configuration>
<property>
<name>year</name>
<value>${coord:formatTime(coord:nominalTime(), "yyyy")}</value>
</property>
<property>
<name>month</name>
<value>${coord:formatTime(coord:nominalTime(), "MM")}</value>
</property>
<property>
<name>date</name>
<value>${coord:formatTime(coord:nominalTime(), "dd")}</value>
</property>
<property>
<name>path_containing_date</name>
<value>${year}/${year}${month}/${year}${month}${date}</value>
</property>
<property>
<name>source_path</name>
<value>${namenodeA_with_parent_path}/${path_containing_date}/${remaining_path}</value>
</property>
<property>
<name>destination_path_1</name>
<value>${namenode1_with_parent_path}/${path_containing_date}/${remaining_path}</value>
</property>
<property>
<name>destination_path_2</name>
<value>${namenode2_with_parent_path}/${path_containing_date}/${remaining_path}</value>
</property>
<property>
<name>destination_path_3</name>
<value>${namenode3_with_parent_path}/${path_containing_date}/${remaining_path}</value>
</property>
</configuration>
This I hope would be more maintainable than using formatTime
everywhere I need date. This way If I want to run manually for a
different date, I can just pass in the year, month and date values.
If this feature is not present right now, Is there a plan to implement it?
On Mon, Oct 19, 2015 at 8:06 AM, Oussama Chougna
<ou...@hotmail.com> wrote:
> Hello Sunil,
> That's not how it works. In the global section you can define oozie properties which are global for all actions in the workflow, like mapred.job.queue.name, job-xml etc..
> What you can do is put 'param1' in your job.properties, then you can resolve it with EL via ${param1}.
>
> Best,
>
> Oussama Chougna
>
>
>> Date: Mon, 19 Oct 2015 07:27:22 -0700
>> Subject: parameter/variable substitution not working in oozie 4.0.0-cdh5.3.2
>> From: bsunil.sp@gmail.com
>> To: user@oozie.apache.org
>>
>> Hi,
>>
>> My Oozie server version: 4.0.0-cdh5.3.2
>> I have a variable A, the value of which would be a combination of
>> other variables (B, C etc.). However, oozie is unable to recognize and
>> substitute values from B, C to form value of A. It throws:
>> javax.servlet.jsp.el.ELException: variable [param1] cannot be resolved
>>
>> Please let me know if it is a known issue in the version I am using.
>>
>> Here is a simple workflow example I am trying:
>> <workflow-app name="test-sunil-wf" xmlns="uri:oozie:workflow:0.4">
>> <global>
>> <configuration>
>> <property>
>> <name>param1</name>
>> <value>idHere</value>
>> </property>
>> </configuration>
>> </global>
>> <start to="test-echo-shell-1"/>
>> <action name="test-echo-shell-1">
>> <shell xmlns="uri:oozie:shell-action:0.1">
>> <job-tracker>${jobTracker}</job-tracker>
>> <name-node>${nameNode}</name-node>
>> <exec>/bin/echo</exec>
>> <argument>${param1}</argument>
>> <capture-output/>
>> </shell>
>> <ok to="end"/>
>> <error to="kill"/>
>> </action>
>> <kill name="kill">
>> <message>Action failed, error
>> message[${wf:errorMessage(wf:lastErrorNode())}]</message>
>> </kill>
>> <end name="end"/>
>> </workflow-app>
>>
>> The above example throws:
>> javax.servlet.jsp.el.ELException: variable [param1] cannot be resolved
>> at org.apache.oozie.util.ELEvaluator$Context.resolveVariable(ELEvaluator.java:106)
>> at org.apache.commons.el.NamedValue.evaluate(NamedValue.java:124)
>> at org.apache.commons.el.ExpressionString.evaluate(ExpressionString.java:114)
>> at org.apache.commons.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:274)
>> at org.apache.commons.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:190)
>> at org.apache.oozie.util.ELEvaluator.evaluate(ELEvaluator.java:203)
>> at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:188)
>> at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:63)
>> at org.apache.oozie.command.XCommand.call(XCommand.java:281)
>> at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:323)
>> at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:252)
>> at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:174)
>>
>>
>> Here is another example which fails. This is somewhat similar to my
>> real use-case:
>> <workflow-app name="test-sunil-wf" xmlns="uri:oozie:workflow:0.4">
>> <start to="test-echo-shell-1"/>
>> <action name="test-echo-shell-1">
>> <shell xmlns="uri:oozie:shell-action:0.1">
>> <job-tracker>${jobTracker}</job-tracker>
>> <name-node>${nameNode}</name-node>
>> <configuration>
>> <property>
>> <name>param1</name>
>> <value>${wf:id()}</value>
>> </property>
>> <property>
>> <name>param2</name>
>> <value>${wf:name()}</value>
>> </property>
>> <property>
>> <name>param3</name>
>> <value>${param1} and ${param2}</value>
>> </property>
>> </configuration>
>> <exec>/bin/echo</exec>
>> <argument>${param3}</argument>
>> <capture-output/>
>> </shell>
>> <ok to="end"/>
>> <error to="kill"/>
>> </action>
>> <kill name="kill">
>> <message>Action failed, error
>> message[${wf:errorMessage(wf:lastErrorNode())}]</message>
>> </kill>
>> <end name="end"/>
>> </workflow-app>
>>
>> The above XML fails with the same stacktrace as shown above.
>>
>> Thanks
>> Sunil
>
RE: parameter/variable substitution not working in oozie
4.0.0-cdh5.3.2
Posted by Oussama Chougna <ou...@hotmail.com>.
Hello Sunil,
That's not how it works. In the global section you can define oozie properties which are global for all actions in the workflow, like mapred.job.queue.name, job-xml etc..
What you can do is put 'param1' in your job.properties, then you can resolve it with EL via ${param1}.
Best,
Oussama Chougna
> Date: Mon, 19 Oct 2015 07:27:22 -0700
> Subject: parameter/variable substitution not working in oozie 4.0.0-cdh5.3.2
> From: bsunil.sp@gmail.com
> To: user@oozie.apache.org
>
> Hi,
>
> My Oozie server version: 4.0.0-cdh5.3.2
> I have a variable A, the value of which would be a combination of
> other variables (B, C etc.). However, oozie is unable to recognize and
> substitute values from B, C to form value of A. It throws:
> javax.servlet.jsp.el.ELException: variable [param1] cannot be resolved
>
> Please let me know if it is a known issue in the version I am using.
>
> Here is a simple workflow example I am trying:
> <workflow-app name="test-sunil-wf" xmlns="uri:oozie:workflow:0.4">
> <global>
> <configuration>
> <property>
> <name>param1</name>
> <value>idHere</value>
> </property>
> </configuration>
> </global>
> <start to="test-echo-shell-1"/>
> <action name="test-echo-shell-1">
> <shell xmlns="uri:oozie:shell-action:0.1">
> <job-tracker>${jobTracker}</job-tracker>
> <name-node>${nameNode}</name-node>
> <exec>/bin/echo</exec>
> <argument>${param1}</argument>
> <capture-output/>
> </shell>
> <ok to="end"/>
> <error to="kill"/>
> </action>
> <kill name="kill">
> <message>Action failed, error
> message[${wf:errorMessage(wf:lastErrorNode())}]</message>
> </kill>
> <end name="end"/>
> </workflow-app>
>
> The above example throws:
> javax.servlet.jsp.el.ELException: variable [param1] cannot be resolved
> at org.apache.oozie.util.ELEvaluator$Context.resolveVariable(ELEvaluator.java:106)
> at org.apache.commons.el.NamedValue.evaluate(NamedValue.java:124)
> at org.apache.commons.el.ExpressionString.evaluate(ExpressionString.java:114)
> at org.apache.commons.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:274)
> at org.apache.commons.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:190)
> at org.apache.oozie.util.ELEvaluator.evaluate(ELEvaluator.java:203)
> at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:188)
> at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:63)
> at org.apache.oozie.command.XCommand.call(XCommand.java:281)
> at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:323)
> at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:252)
> at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:174)
>
>
> Here is another example which fails. This is somewhat similar to my
> real use-case:
> <workflow-app name="test-sunil-wf" xmlns="uri:oozie:workflow:0.4">
> <start to="test-echo-shell-1"/>
> <action name="test-echo-shell-1">
> <shell xmlns="uri:oozie:shell-action:0.1">
> <job-tracker>${jobTracker}</job-tracker>
> <name-node>${nameNode}</name-node>
> <configuration>
> <property>
> <name>param1</name>
> <value>${wf:id()}</value>
> </property>
> <property>
> <name>param2</name>
> <value>${wf:name()}</value>
> </property>
> <property>
> <name>param3</name>
> <value>${param1} and ${param2}</value>
> </property>
> </configuration>
> <exec>/bin/echo</exec>
> <argument>${param3}</argument>
> <capture-output/>
> </shell>
> <ok to="end"/>
> <error to="kill"/>
> </action>
> <kill name="kill">
> <message>Action failed, error
> message[${wf:errorMessage(wf:lastErrorNode())}]</message>
> </kill>
> <end name="end"/>
> </workflow-app>
>
> The above XML fails with the same stacktrace as shown above.
>
> Thanks
> Sunil