Using Oozie to launch a generic Tool

[John Armstrong <jr...@ccri.com>]

Hi everybody.

I'm trying to come up with a good way to launch a generic Tool from Java 
code, using Oozie to send the processing of the Tool out over the 
cluster.  Specifically, I'm writing a Java servlet backend to a webpage, 
which should be able to launch certain M/R Tools on demand.

Currently I'm doing something like this: for each Tool that I want to 
use, I know in advance what properties it will expect in its 
Configuration in order to do its work.  I assemble these, along with a 
pointer to a Tool-specific workflow, into a Properties object called 
"conf" and call OozieClient.run(conf) to submit the job and get back its ID.

The workflow I have set up for each Tool contains a java block like:
<java>
     ...
     <configuration>
         <property>
             <name>my.first.property</name>
             <value>${wf:conf('my.first.property')}</value>
         </property>
         <property>
             <name>my.second.property</name>
             <value>${wf:conf('my.second.property')}</value>
         </property>
         ...
     </configuration>
     <main-class>my.package.and.my.ToolClass</main-class>
     ...
</java>

Of course, this requires knowing all the property names in advance, and 
doesn't generify well.

I'm hoping to write a single workflow file that will work for any of my 
Tools.  AIUI the Properties object I pass to OozieClient.run() 
substitutes for the job.properties file if I launched the Oozie job from 
the CLI (right?)  Is there a method of dumping all the properties from 
this file into the Configuration of the Tool that the Oozie java action 
launches, without knowing the names of the properties in advance?

Thanks for any suggestions you can offer.

Re: Using Oozie to launch a generic Tool

[Harsh J <ha...@cloudera.com>]

John,

Tool-based classes accept -D arguments for config params.

So perhaps adding your java-main workflow with:

<arg>${replaceableargs}</arg>

And then in job.properties:

foo=20
bar=30
replaceableargs=-Dfoo.prop=${foo} -Dbar.prop=${bar}

May work better for you?

On Tue, May 22, 2012 at 4:45 PM, John Armstrong <jr...@ccri.com> wrote:
> Hi everybody.
>
> I'm trying to come up with a good way to launch a generic Tool from Java
> code, using Oozie to send the processing of the Tool out over the cluster.
>  Specifically, I'm writing a Java servlet backend to a webpage, which should
> be able to launch certain M/R Tools on demand.
>
> Currently I'm doing something like this: for each Tool that I want to use, I
> know in advance what properties it will expect in its Configuration in order
> to do its work.  I assemble these, along with a pointer to a Tool-specific
> workflow, into a Properties object called "conf" and call
> OozieClient.run(conf) to submit the job and get back its ID.
>
> The workflow I have set up for each Tool contains a java block like:
> <java>
>    ...
>    <configuration>
>        <property>
>            <name>my.first.property</name>
>            <value>${wf:conf('my.first.property')}</value>
>        </property>
>        <property>
>            <name>my.second.property</name>
>            <value>${wf:conf('my.second.property')}</value>
>        </property>
>        ...
>    </configuration>
>    <main-class>my.package.and.my.ToolClass</main-class>
>    ...
> </java>
>
> Of course, this requires knowing all the property names in advance, and
> doesn't generify well.
>
> I'm hoping to write a single workflow file that will work for any of my
> Tools.  AIUI the Properties object I pass to OozieClient.run() substitutes
> for the job.properties file if I launched the Oozie job from the CLI
> (right?)  Is there a method of dumping all the properties from this file
> into the Configuration of the Tool that the Oozie java action launches,
> without knowing the names of the properties in advance?
>
> Thanks for any suggestions you can offer.



-- 
Harsh J