You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by Mohammad Islam <mi...@yahoo.com.INVALID> on 2014/06/02 20:35:01 UTC
Re: verifying that this is impossible
I'm not sure my last email was delivered.
In my last email, i was proposing to use <parameters> section of workflow to initialize any property once.
Two related links:
1. https://oozie.apache.org/docs/4.0.1/WorkflowFunctionalSpec.html#a4.1_Workflow_Job_Properties_or_Parameters look for formal parameter support in schema 0.4.
2. https://issues.apache.org/jira/browse/OOZIE-239 (Addressed by Robert)
However, I'm not sure if we can specify any EL function as value. Robert can fill it up.
Regards,
Mohammad
On Thursday, May 29, 2014 1:39 PM, Robert Kanter <rk...@cloudera.com> wrote:
The file Oozie prepares is in a standard Hadoop Configuration file format.
e.g.
<configuration>
<property>
<name>some.property.name</name>
<value>some.value</value>
</property>
...
</confguration>
It's location should be stored in an environment variable named
OOZIE_ACTION_CONF_XML
On Thu, May 29, 2014 at 1:28 PM, Scott Preddy <sc...@gmail.com>
wrote:
> Ok, is there a best practice for reading this file oozie prepares? Where is
> it? I don't want to put the el function in the actions because I have many
> of them and would only like to call the el function once. I am calling it
> multiple times now (once for each shell action).
>
>
> On Thu, May 29, 2014 at 3:20 PM, Robert Kanter <rk...@cloudera.com>
> wrote:
>
> > Ah, I see what you're saying. You could have you shell script read the
> > Configuration XML file prepared by Oozie; this would include everything
> in
> > the <configuration> section of the shell action and the <global> section.
> > Though why not simply put the EL Function directly in the shell action
> > instead of a variable?
> >
> >
> >
> >
> > On Thu, May 29, 2014 at 12:23 PM, Scott Preddy <scott.m.preddy@gmail.com
> >
> > wrote:
> >
> > > Right, that what I have experienced. What I would like to know if there
> > is
> > > some mechanism in oozie where I can assign a global variable a value I
> > > obtain from the result of an el function and then have that value
> > > referenced by shell actions?
> > >
> > >
> > > On Thu, May 29, 2014 at 1:02 PM, Robert Kanter <rk...@cloudera.com>
> > > wrote:
> > >
> > > > Hi Scott,
> > > >
> > > > That's not how the <global> seciton works. Properties in the
> > > > <configuration> in the <global> section get applied automatically to
> > all
> > > > <configuration> sections in all actions in the workflow. So, in your
> > > > example, "controlFilePath" is automatically included in your shell
> > > action's
> > > > <configuration> section (even though you omitted it). Putting stuff
> in
> > > the
> > > > <configuration> in the <global> section does not make them a
> variable.
> > > To
> > > > do that, it should be in the job.properties file you use to submit
> the
> > > > workflow. Once you have that, any EL Functions should accept it as
> an
> > > > argument as well.
> > > >
> > > > - Robert
> > > >
> > > >
> > > > On Thu, May 29, 2014 at 8:54 AM, Scott Preddy <
> > scott.m.preddy@gmail.com>
> > > > wrote:
> > > >
> > > > > I would like to set a global variable to my entire oozie job and
> then
> > > > > reference its value from various actions. I would like to use some
> > "el
> > > > > function stuff" when assigning this variable.
> > > > > However I assume this is completely impossible because I cannot
> even
> > do
> > > > the
> > > > > following. No matter what I try "controlFilePath2" is always
> > undefined
> > > in
> > > > > "echoControlFile", so trying the
> > > > > "el function" does not even need to be explored.
> > > > >
> > > > > <workflow-app xmlns="uri:oozie:workflow:0.4" name="sandbox1">
> > > > > <global>
> > > > > <job-tracker>${jobTracker}</job-tracker>
> > > > > <name-node>${nameNode}</name-node>
> > > > > <configuration>
> > > > > <property>
> > > > > <name>mapred.job.queue.name</name>
> > > > > <value>default</value>
> > > > > </property>
> > > > > <property>
> > > > > <name>controlFilePath2</name>
> > > > > <value>***Path***</value>
> > > > > </property>
> > > > > </configuration>
> > > > > </global>
> > > > >
> > > > > <start to="echoControlFile"/>
> > > > >
> > > > > <action name="echoControlFile">
> > > > > <shell xmlns="uri:oozie:shell-action:0.3">
> > > > > <job-tracker>${jobTracker}</job-tracker>
> > > > > <name-node>${nameNode}</name-node>
> > > > > <exec>echo.sh</exec>
> > > > > <argument>Control file path:
> > ${controlFilePath2}</argument>
> > > > > <file>${scriptsDir}/echo.sh#echo.sh</file>
> > > > > <capture-output/>
> > > > > </shell>
> > > > > <ok to="end"/>
> > > > > <error to="fail"/>
> > > > > </action>
> > > > >
> > > > > <kill name="fail">
> > > > > <message>Job failed, error
> > > > > message[${wf:errorMessage(wf:lastErrorNode())}]</message>
> > > > > </kill>
> > > > > <end name="end"/>
> > > > > </workflow-app>
> > > > > "workflow.xml" 53L,
> > > > > 1953C
> > > > > 40,13 Bot
> > > > >
> > > >
> > >
> >
>
Re: verifying that this is impossible
Posted by Robert Kanter <rk...@cloudera.com>.
I forgot about the formal parameters section; I don't remember if it will
accept EL functions.
On Mon, Jun 2, 2014 at 11:35 AM, Mohammad Islam <mi...@yahoo.com.invalid>
wrote:
> I'm not sure my last email was delivered.
> In my last email, i was proposing to use <parameters> section of workflow
> to initialize any property once.
>
> Two related links:
> 1.
> https://oozie.apache.org/docs/4.0.1/WorkflowFunctionalSpec.html#a4.1_Workflow_Job_Properties_or_Parameters
> look for formal parameter support in schema 0.4.
> 2. https://issues.apache.org/jira/browse/OOZIE-239 (Addressed by Robert)
>
> However, I'm not sure if we can specify any EL function as value. Robert
> can fill it up.
>
>
> Regards,
> Mohammad
>
>
>
>
>
>
> On Thursday, May 29, 2014 1:39 PM, Robert Kanter <rk...@cloudera.com>
> wrote:
>
>
>
> The file Oozie prepares is in a standard Hadoop Configuration file format.
> e.g.
> <configuration>
> <property>
> <name>some.property.name</name>
> <value>some.value</value>
> </property>
> ...
> </confguration>
>
> It's location should be stored in an environment variable named
> OOZIE_ACTION_CONF_XML
>
>
>
>
> On Thu, May 29, 2014 at 1:28 PM, Scott Preddy <sc...@gmail.com>
> wrote:
>
> > Ok, is there a best practice for reading this file oozie prepares? Where
> is
> > it? I don't want to put the el function in the actions because I have
> many
> > of them and would only like to call the el function once. I am calling it
> > multiple times now (once for each shell action).
> >
> >
> > On Thu, May 29, 2014 at 3:20 PM, Robert Kanter <rk...@cloudera.com>
> > wrote:
> >
> > > Ah, I see what you're saying. You could have you shell script read the
> > > Configuration XML file prepared by Oozie; this would include everything
> > in
> > > the <configuration> section of the shell action and the <global>
> section.
> > > Though why not simply put the EL Function directly in the shell action
> > > instead of a variable?
> > >
> > >
> > >
> > >
> > > On Thu, May 29, 2014 at 12:23 PM, Scott Preddy <
> scott.m.preddy@gmail.com
> > >
> > > wrote:
> > >
> > > > Right, that what I have experienced. What I would like to know if
> there
> > > is
> > > > some mechanism in oozie where I can assign a global variable a value
> I
> > > > obtain from the result of an el function and then have that value
> > > > referenced by shell actions?
> > > >
> > > >
> > > > On Thu, May 29, 2014 at 1:02 PM, Robert Kanter <rkanter@cloudera.com
> >
> > > > wrote:
> > > >
> > > > > Hi Scott,
> > > > >
> > > > > That's not how the <global> seciton works. Properties in the
> > > > > <configuration> in the <global> section get applied automatically
> to
> > > all
> > > > > <configuration> sections in all actions in the workflow. So, in
> your
> > > > > example, "controlFilePath" is automatically included in your shell
> > > > action's
> > > > > <configuration> section (even though you omitted it). Putting
> stuff
> > in
> > > > the
> > > > > <configuration> in the <global> section does not make them a
> > variable.
> > > > To
> > > > > do that, it should be in the job.properties file you use to submit
> > the
> > > > > workflow. Once you have that, any EL Functions should accept it as
> > an
> > > > > argument as well.
> > > > >
> > > > > - Robert
> > > > >
> > > > >
> > > > > On Thu, May 29, 2014 at 8:54 AM, Scott Preddy <
> > > scott.m.preddy@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > I would like to set a global variable to my entire oozie job and
> > then
> > > > > > reference its value from various actions. I would like to use
> some
> > > "el
> > > > > > function stuff" when assigning this variable.
> > > > > > However I assume this is completely impossible because I cannot
> > even
> > > do
> > > > > the
> > > > > > following. No matter what I try "controlFilePath2" is always
> > > undefined
> > > > in
> > > > > > "echoControlFile", so trying the
> > > > > > "el function" does not even need to be explored.
> > > > > >
> > > > > > <workflow-app xmlns="uri:oozie:workflow:0.4" name="sandbox1">
> > > > > > <global>
> > > > > > <job-tracker>${jobTracker}</job-tracker>
> > > > > > <name-node>${nameNode}</name-node>
> > > > > > <configuration>
> > > > > > <property>
> > > > > > <name>mapred.job.queue.name</name>
> > > > > > <value>default</value>
> > > > > > </property>
> > > > > > <property>
> > > > > > <name>controlFilePath2</name>
> > > > > > <value>***Path***</value>
> > > > > > </property>
> > > > > > </configuration>
> > > > > > </global>
> > > > > >
> > > > > > <start to="echoControlFile"/>
> > > > > >
> > > > > > <action name="echoControlFile">
> > > > > > <shell xmlns="uri:oozie:shell-action:0.3">
> > > > > > <job-tracker>${jobTracker}</job-tracker>
> > > > > > <name-node>${nameNode}</name-node>
> > > > > > <exec>echo.sh</exec>
> > > > > > <argument>Control file path:
> > > ${controlFilePath2}</argument>
> > > > > > <file>${scriptsDir}/echo.sh#echo.sh</file>
> > > > > > <capture-output/>
> > > > > > </shell>
> > > > > > <ok to="end"/>
> > > > > > <error to="fail"/>
> > > > > > </action>
> > > > > >
> > > > > > <kill name="fail">
> > > > > > <message>Job failed, error
> > > > > > message[${wf:errorMessage(wf:lastErrorNode())}]</message>
> > > > > > </kill>
> > > > > > <end name="end"/>
> > > > > > </workflow-app>
> > > > > > "workflow.xml" 53L,
> > > > > > 1953C
> > > > > > 40,13 Bot
> > > > > >
> > > > >
> > > >
> > >
> >
>