You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by Scott Preddy <sc...@gmail.com> on 2014/05/29 17:54:37 UTC
verifying that this is impossible
I would like to set a global variable to my entire oozie job and then
reference its value from various actions. I would like to use some "el
function stuff" when assigning this variable.
However I assume this is completely impossible because I cannot even do the
following. No matter what I try "controlFilePath2" is always undefined in
"echoControlFile", so trying the
"el function" does not even need to be explored.
<workflow-app xmlns="uri:oozie:workflow:0.4" name="sandbox1">
<global>
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>mapred.job.queue.name</name>
<value>default</value>
</property>
<property>
<name>controlFilePath2</name>
<value>***Path***</value>
</property>
</configuration>
</global>
<start to="echoControlFile"/>
<action name="echoControlFile">
<shell xmlns="uri:oozie:shell-action:0.3">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<exec>echo.sh</exec>
<argument>Control file path: ${controlFilePath2}</argument>
<file>${scriptsDir}/echo.sh#echo.sh</file>
<capture-output/>
</shell>
<ok to="end"/>
<error to="fail"/>
</action>
<kill name="fail">
<message>Job failed, error
message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<end name="end"/>
</workflow-app>
"workflow.xml" 53L,
1953C
40,13 Bot
Re: verifying that this is impossible
Posted by Mohammad Islam <mi...@yahoo.com>.
Did you check the parameterization using <parameters> section at the beginning of workflow XML? Any variable can be defined with some default value and can be referred multiple times. http://oozie.apache.org/docs/4.0.1/WorkflowFunctionalSpec.html#a4_Parameterization_of_Workflows
Fine the text "As of schema 0.4, a list of formal parameters can be provided which will allow Oozie to verify, at submission time, that said properties are actually specified (i.e. before the job is executed and fails). Default values can also be provided." and the associated example.
Related JIRA contributed by Robert https://issues.apache.org/jira/browse/OOZIE-239
Regards,
Mohammad
On Thursday, May 29, 2014 1:39 PM, Robert Kanter <rk...@cloudera.com> wrote:
The file Oozie prepares is in a standard Hadoop Configuration file format.
e.g.
<configuration>
<property>
<name>some.property.name</name>
<value>some.value</value>
</property>
...
</confguration>
It's location should be stored in an environment variable named
OOZIE_ACTION_CONF_XML
On Thu, May 29, 2014 at 1:28 PM, Scott Preddy <sc...@gmail.com>
wrote:
> Ok, is there a best practice for reading this file oozie prepares? Where is
> it? I don't want to put the el function in the actions because I have many
> of them and would only like to call the el function once. I am calling it
> multiple times now (once for each shell action).
>
>
> On Thu, May 29, 2014 at 3:20 PM, Robert Kanter <rk...@cloudera.com>
> wrote:
>
> > Ah, I see what you're saying. You could have you shell script read the
> > Configuration XML file prepared by Oozie; this would include everything
> in
> > the <configuration> section of the shell action and the <global> section.
> > Though why not simply put the EL Function directly in the shell action
> > instead of a variable?
> >
> >
> >
> >
> > On Thu, May 29, 2014 at 12:23 PM, Scott Preddy <scott.m.preddy@gmail.com
> >
> > wrote:
> >
> > > Right, that what I have experienced. What I would like to know if there
> > is
> > > some mechanism in oozie where I can assign a global variable a value I
> > > obtain from the result of an el function and then have that value
> > > referenced by shell actions?
> > >
> > >
> > > On Thu, May 29, 2014 at 1:02 PM, Robert Kanter <rk...@cloudera.com>
> > > wrote:
> > >
> > > > Hi Scott,
> > > >
> > > > That's not how the <global> seciton works. Properties in the
> > > > <configuration> in the <global> section get applied automatically to
> > all
> > > > <configuration> sections in all actions in the workflow. So, in your
> > > > example, "controlFilePath" is automatically included in your shell
> > > action's
> > > > <configuration> section (even though you omitted it). Putting stuff
> in
> > > the
> > > > <configuration> in the <global> section does not make them a
> variable.
> > > To
> > > > do that, it should be in the job.properties file you use to submit
> the
> > > > workflow. Once you have that, any EL Functions should accept it as
> an
> > > > argument as well.
> > > >
> > > > - Robert
> > > >
> > > >
> > > > On Thu, May 29, 2014 at 8:54 AM, Scott Preddy <
> > scott.m.preddy@gmail.com>
> > > > wrote:
> > > >
> > > > > I would like to set a global variable to my entire oozie job and
> then
> > > > > reference its value from various actions. I would like to use some
> > "el
> > > > > function stuff" when assigning this variable.
> > > > > However I assume this is completely impossible because I cannot
> even
> > do
> > > > the
> > > > > following. No matter what I try "controlFilePath2" is always
> > undefined
> > > in
> > > > > "echoControlFile", so trying the
> > > > > "el function" does not even need to be explored.
> > > > >
> > > > > <workflow-app xmlns="uri:oozie:workflow:0.4" name="sandbox1">
> > > > > <global>
> > > > > <job-tracker>${jobTracker}</job-tracker>
> > > > > <name-node>${nameNode}</name-node>
> > > > > <configuration>
> > > > > <property>
> > > > > <name>mapred.job.queue.name</name>
> > > > > <value>default</value>
> > > > > </property>
> > > > > <property>
> > > > > <name>controlFilePath2</name>
> > > > > <value>***Path***</value>
> > > > > </property>
> > > > > </configuration>
> > > > > </global>
> > > > >
> > > > > <start to="echoControlFile"/>
> > > > >
> > > > > <action name="echoControlFile">
> > > > > <shell xmlns="uri:oozie:shell-action:0.3">
> > > > > <job-tracker>${jobTracker}</job-tracker>
> > > > > <name-node>${nameNode}</name-node>
> > > > > <exec>echo.sh</exec>
> > > > > <argument>Control file path:
> > ${controlFilePath2}</argument>
> > > > > <file>${scriptsDir}/echo.sh#echo.sh</file>
> > > > > <capture-output/>
> > > > > </shell>
> > > > > <ok to="end"/>
> > > > > <error to="fail"/>
> > > > > </action>
> > > > >
> > > > > <kill name="fail">
> > > > > <message>Job failed, error
> > > > > message[${wf:errorMessage(wf:lastErrorNode())}]</message>
> > > > > </kill>
> > > > > <end name="end"/>
> > > > > </workflow-app>
> > > > > "workflow.xml" 53L,
> > > > > 1953C
> > > > > 40,13 Bot
> > > > >
> > > >
> > >
> >
>
Re: verifying that this is impossible
Posted by Robert Kanter <rk...@cloudera.com>.
I forgot about the formal parameters section; I don't remember if it will
accept EL functions.
On Mon, Jun 2, 2014 at 11:35 AM, Mohammad Islam <mi...@yahoo.com.invalid>
wrote:
> I'm not sure my last email was delivered.
> In my last email, i was proposing to use <parameters> section of workflow
> to initialize any property once.
>
> Two related links:
> 1.
> https://oozie.apache.org/docs/4.0.1/WorkflowFunctionalSpec.html#a4.1_Workflow_Job_Properties_or_Parameters
> look for formal parameter support in schema 0.4.
> 2. https://issues.apache.org/jira/browse/OOZIE-239 (Addressed by Robert)
>
> However, I'm not sure if we can specify any EL function as value. Robert
> can fill it up.
>
>
> Regards,
> Mohammad
>
>
>
>
>
>
> On Thursday, May 29, 2014 1:39 PM, Robert Kanter <rk...@cloudera.com>
> wrote:
>
>
>
> The file Oozie prepares is in a standard Hadoop Configuration file format.
> e.g.
> <configuration>
> <property>
> <name>some.property.name</name>
> <value>some.value</value>
> </property>
> ...
> </confguration>
>
> It's location should be stored in an environment variable named
> OOZIE_ACTION_CONF_XML
>
>
>
>
> On Thu, May 29, 2014 at 1:28 PM, Scott Preddy <sc...@gmail.com>
> wrote:
>
> > Ok, is there a best practice for reading this file oozie prepares? Where
> is
> > it? I don't want to put the el function in the actions because I have
> many
> > of them and would only like to call the el function once. I am calling it
> > multiple times now (once for each shell action).
> >
> >
> > On Thu, May 29, 2014 at 3:20 PM, Robert Kanter <rk...@cloudera.com>
> > wrote:
> >
> > > Ah, I see what you're saying. You could have you shell script read the
> > > Configuration XML file prepared by Oozie; this would include everything
> > in
> > > the <configuration> section of the shell action and the <global>
> section.
> > > Though why not simply put the EL Function directly in the shell action
> > > instead of a variable?
> > >
> > >
> > >
> > >
> > > On Thu, May 29, 2014 at 12:23 PM, Scott Preddy <
> scott.m.preddy@gmail.com
> > >
> > > wrote:
> > >
> > > > Right, that what I have experienced. What I would like to know if
> there
> > > is
> > > > some mechanism in oozie where I can assign a global variable a value
> I
> > > > obtain from the result of an el function and then have that value
> > > > referenced by shell actions?
> > > >
> > > >
> > > > On Thu, May 29, 2014 at 1:02 PM, Robert Kanter <rkanter@cloudera.com
> >
> > > > wrote:
> > > >
> > > > > Hi Scott,
> > > > >
> > > > > That's not how the <global> seciton works. Properties in the
> > > > > <configuration> in the <global> section get applied automatically
> to
> > > all
> > > > > <configuration> sections in all actions in the workflow. So, in
> your
> > > > > example, "controlFilePath" is automatically included in your shell
> > > > action's
> > > > > <configuration> section (even though you omitted it). Putting
> stuff
> > in
> > > > the
> > > > > <configuration> in the <global> section does not make them a
> > variable.
> > > > To
> > > > > do that, it should be in the job.properties file you use to submit
> > the
> > > > > workflow. Once you have that, any EL Functions should accept it as
> > an
> > > > > argument as well.
> > > > >
> > > > > - Robert
> > > > >
> > > > >
> > > > > On Thu, May 29, 2014 at 8:54 AM, Scott Preddy <
> > > scott.m.preddy@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > I would like to set a global variable to my entire oozie job and
> > then
> > > > > > reference its value from various actions. I would like to use
> some
> > > "el
> > > > > > function stuff" when assigning this variable.
> > > > > > However I assume this is completely impossible because I cannot
> > even
> > > do
> > > > > the
> > > > > > following. No matter what I try "controlFilePath2" is always
> > > undefined
> > > > in
> > > > > > "echoControlFile", so trying the
> > > > > > "el function" does not even need to be explored.
> > > > > >
> > > > > > <workflow-app xmlns="uri:oozie:workflow:0.4" name="sandbox1">
> > > > > > <global>
> > > > > > <job-tracker>${jobTracker}</job-tracker>
> > > > > > <name-node>${nameNode}</name-node>
> > > > > > <configuration>
> > > > > > <property>
> > > > > > <name>mapred.job.queue.name</name>
> > > > > > <value>default</value>
> > > > > > </property>
> > > > > > <property>
> > > > > > <name>controlFilePath2</name>
> > > > > > <value>***Path***</value>
> > > > > > </property>
> > > > > > </configuration>
> > > > > > </global>
> > > > > >
> > > > > > <start to="echoControlFile"/>
> > > > > >
> > > > > > <action name="echoControlFile">
> > > > > > <shell xmlns="uri:oozie:shell-action:0.3">
> > > > > > <job-tracker>${jobTracker}</job-tracker>
> > > > > > <name-node>${nameNode}</name-node>
> > > > > > <exec>echo.sh</exec>
> > > > > > <argument>Control file path:
> > > ${controlFilePath2}</argument>
> > > > > > <file>${scriptsDir}/echo.sh#echo.sh</file>
> > > > > > <capture-output/>
> > > > > > </shell>
> > > > > > <ok to="end"/>
> > > > > > <error to="fail"/>
> > > > > > </action>
> > > > > >
> > > > > > <kill name="fail">
> > > > > > <message>Job failed, error
> > > > > > message[${wf:errorMessage(wf:lastErrorNode())}]</message>
> > > > > > </kill>
> > > > > > <end name="end"/>
> > > > > > </workflow-app>
> > > > > > "workflow.xml" 53L,
> > > > > > 1953C
> > > > > > 40,13 Bot
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: verifying that this is impossible
Posted by Mohammad Islam <mi...@yahoo.com.INVALID>.
I'm not sure my last email was delivered.
In my last email, i was proposing to use <parameters> section of workflow to initialize any property once.
Two related links:
1. https://oozie.apache.org/docs/4.0.1/WorkflowFunctionalSpec.html#a4.1_Workflow_Job_Properties_or_Parameters look for formal parameter support in schema 0.4.
2. https://issues.apache.org/jira/browse/OOZIE-239 (Addressed by Robert)
However, I'm not sure if we can specify any EL function as value. Robert can fill it up.
Regards,
Mohammad
On Thursday, May 29, 2014 1:39 PM, Robert Kanter <rk...@cloudera.com> wrote:
The file Oozie prepares is in a standard Hadoop Configuration file format.
e.g.
<configuration>
<property>
<name>some.property.name</name>
<value>some.value</value>
</property>
...
</confguration>
It's location should be stored in an environment variable named
OOZIE_ACTION_CONF_XML
On Thu, May 29, 2014 at 1:28 PM, Scott Preddy <sc...@gmail.com>
wrote:
> Ok, is there a best practice for reading this file oozie prepares? Where is
> it? I don't want to put the el function in the actions because I have many
> of them and would only like to call the el function once. I am calling it
> multiple times now (once for each shell action).
>
>
> On Thu, May 29, 2014 at 3:20 PM, Robert Kanter <rk...@cloudera.com>
> wrote:
>
> > Ah, I see what you're saying. You could have you shell script read the
> > Configuration XML file prepared by Oozie; this would include everything
> in
> > the <configuration> section of the shell action and the <global> section.
> > Though why not simply put the EL Function directly in the shell action
> > instead of a variable?
> >
> >
> >
> >
> > On Thu, May 29, 2014 at 12:23 PM, Scott Preddy <scott.m.preddy@gmail.com
> >
> > wrote:
> >
> > > Right, that what I have experienced. What I would like to know if there
> > is
> > > some mechanism in oozie where I can assign a global variable a value I
> > > obtain from the result of an el function and then have that value
> > > referenced by shell actions?
> > >
> > >
> > > On Thu, May 29, 2014 at 1:02 PM, Robert Kanter <rk...@cloudera.com>
> > > wrote:
> > >
> > > > Hi Scott,
> > > >
> > > > That's not how the <global> seciton works. Properties in the
> > > > <configuration> in the <global> section get applied automatically to
> > all
> > > > <configuration> sections in all actions in the workflow. So, in your
> > > > example, "controlFilePath" is automatically included in your shell
> > > action's
> > > > <configuration> section (even though you omitted it). Putting stuff
> in
> > > the
> > > > <configuration> in the <global> section does not make them a
> variable.
> > > To
> > > > do that, it should be in the job.properties file you use to submit
> the
> > > > workflow. Once you have that, any EL Functions should accept it as
> an
> > > > argument as well.
> > > >
> > > > - Robert
> > > >
> > > >
> > > > On Thu, May 29, 2014 at 8:54 AM, Scott Preddy <
> > scott.m.preddy@gmail.com>
> > > > wrote:
> > > >
> > > > > I would like to set a global variable to my entire oozie job and
> then
> > > > > reference its value from various actions. I would like to use some
> > "el
> > > > > function stuff" when assigning this variable.
> > > > > However I assume this is completely impossible because I cannot
> even
> > do
> > > > the
> > > > > following. No matter what I try "controlFilePath2" is always
> > undefined
> > > in
> > > > > "echoControlFile", so trying the
> > > > > "el function" does not even need to be explored.
> > > > >
> > > > > <workflow-app xmlns="uri:oozie:workflow:0.4" name="sandbox1">
> > > > > <global>
> > > > > <job-tracker>${jobTracker}</job-tracker>
> > > > > <name-node>${nameNode}</name-node>
> > > > > <configuration>
> > > > > <property>
> > > > > <name>mapred.job.queue.name</name>
> > > > > <value>default</value>
> > > > > </property>
> > > > > <property>
> > > > > <name>controlFilePath2</name>
> > > > > <value>***Path***</value>
> > > > > </property>
> > > > > </configuration>
> > > > > </global>
> > > > >
> > > > > <start to="echoControlFile"/>
> > > > >
> > > > > <action name="echoControlFile">
> > > > > <shell xmlns="uri:oozie:shell-action:0.3">
> > > > > <job-tracker>${jobTracker}</job-tracker>
> > > > > <name-node>${nameNode}</name-node>
> > > > > <exec>echo.sh</exec>
> > > > > <argument>Control file path:
> > ${controlFilePath2}</argument>
> > > > > <file>${scriptsDir}/echo.sh#echo.sh</file>
> > > > > <capture-output/>
> > > > > </shell>
> > > > > <ok to="end"/>
> > > > > <error to="fail"/>
> > > > > </action>
> > > > >
> > > > > <kill name="fail">
> > > > > <message>Job failed, error
> > > > > message[${wf:errorMessage(wf:lastErrorNode())}]</message>
> > > > > </kill>
> > > > > <end name="end"/>
> > > > > </workflow-app>
> > > > > "workflow.xml" 53L,
> > > > > 1953C
> > > > > 40,13 Bot
> > > > >
> > > >
> > >
> >
>
Re: verifying that this is impossible
Posted by Robert Kanter <rk...@cloudera.com>.
The file Oozie prepares is in a standard Hadoop Configuration file format.
e.g.
<configuration>
<property>
<name>some.property.name</name>
<value>some.value</value>
</property>
...
</confguration>
It's location should be stored in an environment variable named
OOZIE_ACTION_CONF_XML
On Thu, May 29, 2014 at 1:28 PM, Scott Preddy <sc...@gmail.com>
wrote:
> Ok, is there a best practice for reading this file oozie prepares? Where is
> it? I don't want to put the el function in the actions because I have many
> of them and would only like to call the el function once. I am calling it
> multiple times now (once for each shell action).
>
>
> On Thu, May 29, 2014 at 3:20 PM, Robert Kanter <rk...@cloudera.com>
> wrote:
>
> > Ah, I see what you're saying. You could have you shell script read the
> > Configuration XML file prepared by Oozie; this would include everything
> in
> > the <configuration> section of the shell action and the <global> section.
> > Though why not simply put the EL Function directly in the shell action
> > instead of a variable?
> >
> >
> >
> >
> > On Thu, May 29, 2014 at 12:23 PM, Scott Preddy <scott.m.preddy@gmail.com
> >
> > wrote:
> >
> > > Right, that what I have experienced. What I would like to know if there
> > is
> > > some mechanism in oozie where I can assign a global variable a value I
> > > obtain from the result of an el function and then have that value
> > > referenced by shell actions?
> > >
> > >
> > > On Thu, May 29, 2014 at 1:02 PM, Robert Kanter <rk...@cloudera.com>
> > > wrote:
> > >
> > > > Hi Scott,
> > > >
> > > > That's not how the <global> seciton works. Properties in the
> > > > <configuration> in the <global> section get applied automatically to
> > all
> > > > <configuration> sections in all actions in the workflow. So, in your
> > > > example, "controlFilePath" is automatically included in your shell
> > > action's
> > > > <configuration> section (even though you omitted it). Putting stuff
> in
> > > the
> > > > <configuration> in the <global> section does not make them a
> variable.
> > > To
> > > > do that, it should be in the job.properties file you use to submit
> the
> > > > workflow. Once you have that, any EL Functions should accept it as
> an
> > > > argument as well.
> > > >
> > > > - Robert
> > > >
> > > >
> > > > On Thu, May 29, 2014 at 8:54 AM, Scott Preddy <
> > scott.m.preddy@gmail.com>
> > > > wrote:
> > > >
> > > > > I would like to set a global variable to my entire oozie job and
> then
> > > > > reference its value from various actions. I would like to use some
> > "el
> > > > > function stuff" when assigning this variable.
> > > > > However I assume this is completely impossible because I cannot
> even
> > do
> > > > the
> > > > > following. No matter what I try "controlFilePath2" is always
> > undefined
> > > in
> > > > > "echoControlFile", so trying the
> > > > > "el function" does not even need to be explored.
> > > > >
> > > > > <workflow-app xmlns="uri:oozie:workflow:0.4" name="sandbox1">
> > > > > <global>
> > > > > <job-tracker>${jobTracker}</job-tracker>
> > > > > <name-node>${nameNode}</name-node>
> > > > > <configuration>
> > > > > <property>
> > > > > <name>mapred.job.queue.name</name>
> > > > > <value>default</value>
> > > > > </property>
> > > > > <property>
> > > > > <name>controlFilePath2</name>
> > > > > <value>***Path***</value>
> > > > > </property>
> > > > > </configuration>
> > > > > </global>
> > > > >
> > > > > <start to="echoControlFile"/>
> > > > >
> > > > > <action name="echoControlFile">
> > > > > <shell xmlns="uri:oozie:shell-action:0.3">
> > > > > <job-tracker>${jobTracker}</job-tracker>
> > > > > <name-node>${nameNode}</name-node>
> > > > > <exec>echo.sh</exec>
> > > > > <argument>Control file path:
> > ${controlFilePath2}</argument>
> > > > > <file>${scriptsDir}/echo.sh#echo.sh</file>
> > > > > <capture-output/>
> > > > > </shell>
> > > > > <ok to="end"/>
> > > > > <error to="fail"/>
> > > > > </action>
> > > > >
> > > > > <kill name="fail">
> > > > > <message>Job failed, error
> > > > > message[${wf:errorMessage(wf:lastErrorNode())}]</message>
> > > > > </kill>
> > > > > <end name="end"/>
> > > > > </workflow-app>
> > > > > "workflow.xml" 53L,
> > > > > 1953C
> > > > > 40,13 Bot
> > > > >
> > > >
> > >
> >
>
Re: verifying that this is impossible
Posted by Scott Preddy <sc...@gmail.com>.
Ok, is there a best practice for reading this file oozie prepares? Where is
it? I don't want to put the el function in the actions because I have many
of them and would only like to call the el function once. I am calling it
multiple times now (once for each shell action).
On Thu, May 29, 2014 at 3:20 PM, Robert Kanter <rk...@cloudera.com> wrote:
> Ah, I see what you're saying. You could have you shell script read the
> Configuration XML file prepared by Oozie; this would include everything in
> the <configuration> section of the shell action and the <global> section.
> Though why not simply put the EL Function directly in the shell action
> instead of a variable?
>
>
>
>
> On Thu, May 29, 2014 at 12:23 PM, Scott Preddy <sc...@gmail.com>
> wrote:
>
> > Right, that what I have experienced. What I would like to know if there
> is
> > some mechanism in oozie where I can assign a global variable a value I
> > obtain from the result of an el function and then have that value
> > referenced by shell actions?
> >
> >
> > On Thu, May 29, 2014 at 1:02 PM, Robert Kanter <rk...@cloudera.com>
> > wrote:
> >
> > > Hi Scott,
> > >
> > > That's not how the <global> seciton works. Properties in the
> > > <configuration> in the <global> section get applied automatically to
> all
> > > <configuration> sections in all actions in the workflow. So, in your
> > > example, "controlFilePath" is automatically included in your shell
> > action's
> > > <configuration> section (even though you omitted it). Putting stuff in
> > the
> > > <configuration> in the <global> section does not make them a variable.
> > To
> > > do that, it should be in the job.properties file you use to submit the
> > > workflow. Once you have that, any EL Functions should accept it as an
> > > argument as well.
> > >
> > > - Robert
> > >
> > >
> > > On Thu, May 29, 2014 at 8:54 AM, Scott Preddy <
> scott.m.preddy@gmail.com>
> > > wrote:
> > >
> > > > I would like to set a global variable to my entire oozie job and then
> > > > reference its value from various actions. I would like to use some
> "el
> > > > function stuff" when assigning this variable.
> > > > However I assume this is completely impossible because I cannot even
> do
> > > the
> > > > following. No matter what I try "controlFilePath2" is always
> undefined
> > in
> > > > "echoControlFile", so trying the
> > > > "el function" does not even need to be explored.
> > > >
> > > > <workflow-app xmlns="uri:oozie:workflow:0.4" name="sandbox1">
> > > > <global>
> > > > <job-tracker>${jobTracker}</job-tracker>
> > > > <name-node>${nameNode}</name-node>
> > > > <configuration>
> > > > <property>
> > > > <name>mapred.job.queue.name</name>
> > > > <value>default</value>
> > > > </property>
> > > > <property>
> > > > <name>controlFilePath2</name>
> > > > <value>***Path***</value>
> > > > </property>
> > > > </configuration>
> > > > </global>
> > > >
> > > > <start to="echoControlFile"/>
> > > >
> > > > <action name="echoControlFile">
> > > > <shell xmlns="uri:oozie:shell-action:0.3">
> > > > <job-tracker>${jobTracker}</job-tracker>
> > > > <name-node>${nameNode}</name-node>
> > > > <exec>echo.sh</exec>
> > > > <argument>Control file path:
> ${controlFilePath2}</argument>
> > > > <file>${scriptsDir}/echo.sh#echo.sh</file>
> > > > <capture-output/>
> > > > </shell>
> > > > <ok to="end"/>
> > > > <error to="fail"/>
> > > > </action>
> > > >
> > > > <kill name="fail">
> > > > <message>Job failed, error
> > > > message[${wf:errorMessage(wf:lastErrorNode())}]</message>
> > > > </kill>
> > > > <end name="end"/>
> > > > </workflow-app>
> > > > "workflow.xml" 53L,
> > > > 1953C
> > > > 40,13 Bot
> > > >
> > >
> >
>
Re: verifying that this is impossible
Posted by Robert Kanter <rk...@cloudera.com>.
Ah, I see what you're saying. You could have you shell script read the
Configuration XML file prepared by Oozie; this would include everything in
the <configuration> section of the shell action and the <global> section.
Though why not simply put the EL Function directly in the shell action
instead of a variable?
On Thu, May 29, 2014 at 12:23 PM, Scott Preddy <sc...@gmail.com>
wrote:
> Right, that what I have experienced. What I would like to know if there is
> some mechanism in oozie where I can assign a global variable a value I
> obtain from the result of an el function and then have that value
> referenced by shell actions?
>
>
> On Thu, May 29, 2014 at 1:02 PM, Robert Kanter <rk...@cloudera.com>
> wrote:
>
> > Hi Scott,
> >
> > That's not how the <global> seciton works. Properties in the
> > <configuration> in the <global> section get applied automatically to all
> > <configuration> sections in all actions in the workflow. So, in your
> > example, "controlFilePath" is automatically included in your shell
> action's
> > <configuration> section (even though you omitted it). Putting stuff in
> the
> > <configuration> in the <global> section does not make them a variable.
> To
> > do that, it should be in the job.properties file you use to submit the
> > workflow. Once you have that, any EL Functions should accept it as an
> > argument as well.
> >
> > - Robert
> >
> >
> > On Thu, May 29, 2014 at 8:54 AM, Scott Preddy <sc...@gmail.com>
> > wrote:
> >
> > > I would like to set a global variable to my entire oozie job and then
> > > reference its value from various actions. I would like to use some "el
> > > function stuff" when assigning this variable.
> > > However I assume this is completely impossible because I cannot even do
> > the
> > > following. No matter what I try "controlFilePath2" is always undefined
> in
> > > "echoControlFile", so trying the
> > > "el function" does not even need to be explored.
> > >
> > > <workflow-app xmlns="uri:oozie:workflow:0.4" name="sandbox1">
> > > <global>
> > > <job-tracker>${jobTracker}</job-tracker>
> > > <name-node>${nameNode}</name-node>
> > > <configuration>
> > > <property>
> > > <name>mapred.job.queue.name</name>
> > > <value>default</value>
> > > </property>
> > > <property>
> > > <name>controlFilePath2</name>
> > > <value>***Path***</value>
> > > </property>
> > > </configuration>
> > > </global>
> > >
> > > <start to="echoControlFile"/>
> > >
> > > <action name="echoControlFile">
> > > <shell xmlns="uri:oozie:shell-action:0.3">
> > > <job-tracker>${jobTracker}</job-tracker>
> > > <name-node>${nameNode}</name-node>
> > > <exec>echo.sh</exec>
> > > <argument>Control file path: ${controlFilePath2}</argument>
> > > <file>${scriptsDir}/echo.sh#echo.sh</file>
> > > <capture-output/>
> > > </shell>
> > > <ok to="end"/>
> > > <error to="fail"/>
> > > </action>
> > >
> > > <kill name="fail">
> > > <message>Job failed, error
> > > message[${wf:errorMessage(wf:lastErrorNode())}]</message>
> > > </kill>
> > > <end name="end"/>
> > > </workflow-app>
> > > "workflow.xml" 53L,
> > > 1953C
> > > 40,13 Bot
> > >
> >
>
Re: verifying that this is impossible
Posted by Scott Preddy <sc...@gmail.com>.
Right, that what I have experienced. What I would like to know if there is
some mechanism in oozie where I can assign a global variable a value I
obtain from the result of an el function and then have that value
referenced by shell actions?
On Thu, May 29, 2014 at 1:02 PM, Robert Kanter <rk...@cloudera.com> wrote:
> Hi Scott,
>
> That's not how the <global> seciton works. Properties in the
> <configuration> in the <global> section get applied automatically to all
> <configuration> sections in all actions in the workflow. So, in your
> example, "controlFilePath" is automatically included in your shell action's
> <configuration> section (even though you omitted it). Putting stuff in the
> <configuration> in the <global> section does not make them a variable. To
> do that, it should be in the job.properties file you use to submit the
> workflow. Once you have that, any EL Functions should accept it as an
> argument as well.
>
> - Robert
>
>
> On Thu, May 29, 2014 at 8:54 AM, Scott Preddy <sc...@gmail.com>
> wrote:
>
> > I would like to set a global variable to my entire oozie job and then
> > reference its value from various actions. I would like to use some "el
> > function stuff" when assigning this variable.
> > However I assume this is completely impossible because I cannot even do
> the
> > following. No matter what I try "controlFilePath2" is always undefined in
> > "echoControlFile", so trying the
> > "el function" does not even need to be explored.
> >
> > <workflow-app xmlns="uri:oozie:workflow:0.4" name="sandbox1">
> > <global>
> > <job-tracker>${jobTracker}</job-tracker>
> > <name-node>${nameNode}</name-node>
> > <configuration>
> > <property>
> > <name>mapred.job.queue.name</name>
> > <value>default</value>
> > </property>
> > <property>
> > <name>controlFilePath2</name>
> > <value>***Path***</value>
> > </property>
> > </configuration>
> > </global>
> >
> > <start to="echoControlFile"/>
> >
> > <action name="echoControlFile">
> > <shell xmlns="uri:oozie:shell-action:0.3">
> > <job-tracker>${jobTracker}</job-tracker>
> > <name-node>${nameNode}</name-node>
> > <exec>echo.sh</exec>
> > <argument>Control file path: ${controlFilePath2}</argument>
> > <file>${scriptsDir}/echo.sh#echo.sh</file>
> > <capture-output/>
> > </shell>
> > <ok to="end"/>
> > <error to="fail"/>
> > </action>
> >
> > <kill name="fail">
> > <message>Job failed, error
> > message[${wf:errorMessage(wf:lastErrorNode())}]</message>
> > </kill>
> > <end name="end"/>
> > </workflow-app>
> > "workflow.xml" 53L,
> > 1953C
> > 40,13 Bot
> >
>
Re: verifying that this is impossible
Posted by Robert Kanter <rk...@cloudera.com>.
Hi Scott,
That's not how the <global> seciton works. Properties in the
<configuration> in the <global> section get applied automatically to all
<configuration> sections in all actions in the workflow. So, in your
example, "controlFilePath" is automatically included in your shell action's
<configuration> section (even though you omitted it). Putting stuff in the
<configuration> in the <global> section does not make them a variable. To
do that, it should be in the job.properties file you use to submit the
workflow. Once you have that, any EL Functions should accept it as an
argument as well.
- Robert
On Thu, May 29, 2014 at 8:54 AM, Scott Preddy <sc...@gmail.com>
wrote:
> I would like to set a global variable to my entire oozie job and then
> reference its value from various actions. I would like to use some "el
> function stuff" when assigning this variable.
> However I assume this is completely impossible because I cannot even do the
> following. No matter what I try "controlFilePath2" is always undefined in
> "echoControlFile", so trying the
> "el function" does not even need to be explored.
>
> <workflow-app xmlns="uri:oozie:workflow:0.4" name="sandbox1">
> <global>
> <job-tracker>${jobTracker}</job-tracker>
> <name-node>${nameNode}</name-node>
> <configuration>
> <property>
> <name>mapred.job.queue.name</name>
> <value>default</value>
> </property>
> <property>
> <name>controlFilePath2</name>
> <value>***Path***</value>
> </property>
> </configuration>
> </global>
>
> <start to="echoControlFile"/>
>
> <action name="echoControlFile">
> <shell xmlns="uri:oozie:shell-action:0.3">
> <job-tracker>${jobTracker}</job-tracker>
> <name-node>${nameNode}</name-node>
> <exec>echo.sh</exec>
> <argument>Control file path: ${controlFilePath2}</argument>
> <file>${scriptsDir}/echo.sh#echo.sh</file>
> <capture-output/>
> </shell>
> <ok to="end"/>
> <error to="fail"/>
> </action>
>
> <kill name="fail">
> <message>Job failed, error
> message[${wf:errorMessage(wf:lastErrorNode())}]</message>
> </kill>
> <end name="end"/>
> </workflow-app>
> "workflow.xml" 53L,
> 1953C
> 40,13 Bot
>