You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by Serega Sheypak <se...@gmail.com> on 2016/04/24 22:51:12 UTC
Is there any way to specify custom lib dir for each workflow action
Hi, oozie workflow by default expects special file layout:
The directory structure looks like this:
- wf-app-dir/workflow.xml
- wf-app-dir/lib
- wf-app-dir/lib/myJavaClasses.JAR
Is there any way to specify custom wf--app-dir/lib for each oozie workflow
action? My workflow actions are java-actions and I want to avoid jar-hell
in /lib dir common for the whole workflow.
Re: Is there any way to specify custom lib dir for each workflow action
Posted by Serega Sheypak <se...@gmail.com>.
Hm... But I can override oozie.libpath... Why I can't get benefit from both
properties? My goal is to make isolated class path for each Java action.
понедельник, 25 апреля 2016 г. пользователь Abhishek Bafna написал:
> Hi,
>
> It works little different than you expect. If you want to use the
> oozie.action.sharelib.for.java then create those directories under
> /user/oozie/share/lib/lib_*/<here>.
>
> Thanks,
> Abhishek
>
> On Mon, Apr 25, 2016 at 9:12 PM, Serega Sheypak <serega.sheypak@gmail.com
> <javascript:;>>
> wrote:
>
> > Hi, thanks, I know about archives option
> >
> >
> http://blog.cloudera.com/blog/2014/05/how-to-use-the-sharelib-in-apache-oozie-cdh-5/
> >
> > says it's possible to override *oozie.libpath *and
> > *oozie.action.sharelib.for.java*
> >
> > I have layout on hdfs:
> > /user/me/ddj-workflow/java/java-ddj-228/lib
> > /user/me/ddj-workflow/java/java-ddj-227/lib
> >
> > Both catalogs contain diffrent set of jar.
> > I try to override during workflow submission:
> > *oozie.libpath=/user/me/libs/ddj-workflow*
> >
> >
> > and I set properties inside action definition:
> >
> > <action name="ddj-280-DumpJsonJob">
> > <java>
> > <job-tracker>${jobTracker}</job-tracker>
> > <name-node>${nameNode}</name-node>
> > <configuration>
> > * <property>*
> > * <name>oozie.action.sharelib.for.java</name>*
> > * <value>java-ddj-228</value> *
> > * </property>*
> > </configuration>
> > <main-class>com.twitter.scalding.Tool</main-class>
> > <arg>my.company.Class</arg>
> > <arg>--input input</arg>
> > <arg>--output output</arg>
> > </java>
> > <ok to="ddj-270-DumpJsonJob"/>
> > <error to="kill"/>
> > </action>
> >
> > It doesn't work, files from /user/me/ddj-workflow/java/java-ddj-228/lib
> > are not in classpath.
> >
> >
> >
> >
> >
> > 2016-04-25 16:53 GMT+02:00 Abhishek Bafna <bafna.iitr@gmail.com
> <javascript:;>>:
> >
> > > Hi,
> > >
> > > Currently, there is not way specify the separate wf-app-dir/lib for
> > > individual actions. But you can use the file and archive elements for
> > each
> > > java action.
> > >
> > >
> > >
> >
> https://oozie.apache.org/docs/4.2.0/WorkflowFunctionalSpec.html#a3.2.2.1_Adding_Files_and_Archives_for_the_Job
> > >
> > > Thanks,
> > > Abhishek
> > >
> > > On Mon, Apr 25, 2016 at 2:21 AM, Serega Sheypak <
> > serega.sheypak@gmail.com <javascript:;>>
> > > wrote:
> > >
> > > > Hi, oozie workflow by default expects special file layout:
> > > >
> > > > The directory structure looks like this:
> > > >
> > > > - wf-app-dir/workflow.xml
> > > > - wf-app-dir/lib
> > > > - wf-app-dir/lib/myJavaClasses.JAR
> > > >
> > > >
> > > > Is there any way to specify custom wf--app-dir/lib for each oozie
> > > workflow
> > > > action? My workflow actions are java-actions and I want to avoid
> > jar-hell
> > > > in /lib dir common for the whole workflow.
> > > >
> > >
> >
>
Re: Is there any way to specify custom lib dir for each workflow action
Posted by Serega Sheypak <se...@gmail.com>.
oozie.launcher.oozie.libpath
set per-action works fine for me! I can set isolated classpath for every
java-action and avoid jar-hell.
2016-04-25 17:55 GMT+02:00 Abhishek Bafna <ba...@gmail.com>:
> Hi,
>
> It works little different than you expect. If you want to use the
> oozie.action.sharelib.for.java then create those directories under
> /user/oozie/share/lib/lib_*/<here>.
>
> Thanks,
> Abhishek
>
> On Mon, Apr 25, 2016 at 9:12 PM, Serega Sheypak <se...@gmail.com>
> wrote:
>
> > Hi, thanks, I know about archives option
> >
> >
> http://blog.cloudera.com/blog/2014/05/how-to-use-the-sharelib-in-apache-oozie-cdh-5/
> >
> > says it's possible to override *oozie.libpath *and
> > *oozie.action.sharelib.for.java*
> >
> > I have layout on hdfs:
> > /user/me/ddj-workflow/java/java-ddj-228/lib
> > /user/me/ddj-workflow/java/java-ddj-227/lib
> >
> > Both catalogs contain diffrent set of jar.
> > I try to override during workflow submission:
> > *oozie.libpath=/user/me/libs/ddj-workflow*
> >
> >
> > and I set properties inside action definition:
> >
> > <action name="ddj-280-DumpJsonJob">
> > <java>
> > <job-tracker>${jobTracker}</job-tracker>
> > <name-node>${nameNode}</name-node>
> > <configuration>
> > * <property>*
> > * <name>oozie.action.sharelib.for.java</name>*
> > * <value>java-ddj-228</value> *
> > * </property>*
> > </configuration>
> > <main-class>com.twitter.scalding.Tool</main-class>
> > <arg>my.company.Class</arg>
> > <arg>--input input</arg>
> > <arg>--output output</arg>
> > </java>
> > <ok to="ddj-270-DumpJsonJob"/>
> > <error to="kill"/>
> > </action>
> >
> > It doesn't work, files from /user/me/ddj-workflow/java/java-ddj-228/lib
> > are not in classpath.
> >
> >
> >
> >
> >
> > 2016-04-25 16:53 GMT+02:00 Abhishek Bafna <ba...@gmail.com>:
> >
> > > Hi,
> > >
> > > Currently, there is not way specify the separate wf-app-dir/lib for
> > > individual actions. But you can use the file and archive elements for
> > each
> > > java action.
> > >
> > >
> > >
> >
> https://oozie.apache.org/docs/4.2.0/WorkflowFunctionalSpec.html#a3.2.2.1_Adding_Files_and_Archives_for_the_Job
> > >
> > > Thanks,
> > > Abhishek
> > >
> > > On Mon, Apr 25, 2016 at 2:21 AM, Serega Sheypak <
> > serega.sheypak@gmail.com>
> > > wrote:
> > >
> > > > Hi, oozie workflow by default expects special file layout:
> > > >
> > > > The directory structure looks like this:
> > > >
> > > > - wf-app-dir/workflow.xml
> > > > - wf-app-dir/lib
> > > > - wf-app-dir/lib/myJavaClasses.JAR
> > > >
> > > >
> > > > Is there any way to specify custom wf--app-dir/lib for each oozie
> > > workflow
> > > > action? My workflow actions are java-actions and I want to avoid
> > jar-hell
> > > > in /lib dir common for the whole workflow.
> > > >
> > >
> >
>
Re: Is there any way to specify custom lib dir for each workflow action
Posted by Abhishek Bafna <ba...@gmail.com>.
Hi,
It works little different than you expect. If you want to use the
oozie.action.sharelib.for.java then create those directories under
/user/oozie/share/lib/lib_*/<here>.
Thanks,
Abhishek
On Mon, Apr 25, 2016 at 9:12 PM, Serega Sheypak <se...@gmail.com>
wrote:
> Hi, thanks, I know about archives option
>
> http://blog.cloudera.com/blog/2014/05/how-to-use-the-sharelib-in-apache-oozie-cdh-5/
>
> says it's possible to override *oozie.libpath *and
> *oozie.action.sharelib.for.java*
>
> I have layout on hdfs:
> /user/me/ddj-workflow/java/java-ddj-228/lib
> /user/me/ddj-workflow/java/java-ddj-227/lib
>
> Both catalogs contain diffrent set of jar.
> I try to override during workflow submission:
> *oozie.libpath=/user/me/libs/ddj-workflow*
>
>
> and I set properties inside action definition:
>
> <action name="ddj-280-DumpJsonJob">
> <java>
> <job-tracker>${jobTracker}</job-tracker>
> <name-node>${nameNode}</name-node>
> <configuration>
> * <property>*
> * <name>oozie.action.sharelib.for.java</name>*
> * <value>java-ddj-228</value> *
> * </property>*
> </configuration>
> <main-class>com.twitter.scalding.Tool</main-class>
> <arg>my.company.Class</arg>
> <arg>--input input</arg>
> <arg>--output output</arg>
> </java>
> <ok to="ddj-270-DumpJsonJob"/>
> <error to="kill"/>
> </action>
>
> It doesn't work, files from /user/me/ddj-workflow/java/java-ddj-228/lib
> are not in classpath.
>
>
>
>
>
> 2016-04-25 16:53 GMT+02:00 Abhishek Bafna <ba...@gmail.com>:
>
> > Hi,
> >
> > Currently, there is not way specify the separate wf-app-dir/lib for
> > individual actions. But you can use the file and archive elements for
> each
> > java action.
> >
> >
> >
> https://oozie.apache.org/docs/4.2.0/WorkflowFunctionalSpec.html#a3.2.2.1_Adding_Files_and_Archives_for_the_Job
> >
> > Thanks,
> > Abhishek
> >
> > On Mon, Apr 25, 2016 at 2:21 AM, Serega Sheypak <
> serega.sheypak@gmail.com>
> > wrote:
> >
> > > Hi, oozie workflow by default expects special file layout:
> > >
> > > The directory structure looks like this:
> > >
> > > - wf-app-dir/workflow.xml
> > > - wf-app-dir/lib
> > > - wf-app-dir/lib/myJavaClasses.JAR
> > >
> > >
> > > Is there any way to specify custom wf--app-dir/lib for each oozie
> > workflow
> > > action? My workflow actions are java-actions and I want to avoid
> jar-hell
> > > in /lib dir common for the whole workflow.
> > >
> >
>
Re: Is there any way to specify custom lib dir for each workflow action
Posted by Serega Sheypak <se...@gmail.com>.
Hi, thanks, I know about archives option
http://blog.cloudera.com/blog/2014/05/how-to-use-the-sharelib-in-apache-oozie-cdh-5/
says it's possible to override *oozie.libpath *and
*oozie.action.sharelib.for.java*
I have layout on hdfs:
/user/me/ddj-workflow/java/java-ddj-228/lib
/user/me/ddj-workflow/java/java-ddj-227/lib
Both catalogs contain diffrent set of jar.
I try to override during workflow submission:
*oozie.libpath=/user/me/libs/ddj-workflow*
and I set properties inside action definition:
<action name="ddj-280-DumpJsonJob">
<java>
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
* <property>*
* <name>oozie.action.sharelib.for.java</name>*
* <value>java-ddj-228</value> *
* </property>*
</configuration>
<main-class>com.twitter.scalding.Tool</main-class>
<arg>my.company.Class</arg>
<arg>--input input</arg>
<arg>--output output</arg>
</java>
<ok to="ddj-270-DumpJsonJob"/>
<error to="kill"/>
</action>
It doesn't work, files from /user/me/ddj-workflow/java/java-ddj-228/lib
are not in classpath.
2016-04-25 16:53 GMT+02:00 Abhishek Bafna <ba...@gmail.com>:
> Hi,
>
> Currently, there is not way specify the separate wf-app-dir/lib for
> individual actions. But you can use the file and archive elements for each
> java action.
>
>
> https://oozie.apache.org/docs/4.2.0/WorkflowFunctionalSpec.html#a3.2.2.1_Adding_Files_and_Archives_for_the_Job
>
> Thanks,
> Abhishek
>
> On Mon, Apr 25, 2016 at 2:21 AM, Serega Sheypak <se...@gmail.com>
> wrote:
>
> > Hi, oozie workflow by default expects special file layout:
> >
> > The directory structure looks like this:
> >
> > - wf-app-dir/workflow.xml
> > - wf-app-dir/lib
> > - wf-app-dir/lib/myJavaClasses.JAR
> >
> >
> > Is there any way to specify custom wf--app-dir/lib for each oozie
> workflow
> > action? My workflow actions are java-actions and I want to avoid jar-hell
> > in /lib dir common for the whole workflow.
> >
>
Re: Is there any way to specify custom lib dir for each workflow action
Posted by Amit Kumar <ak...@gmail.com>.
unsubscribe
On Mon, Apr 25, 2016 at 8:23 PM, Abhishek Bafna <ba...@gmail.com>
wrote:
> Hi,
>
> Currently, there is not way specify the separate wf-app-dir/lib for
> individual actions. But you can use the file and archive elements for each
> java action.
>
>
> https://oozie.apache.org/docs/4.2.0/WorkflowFunctionalSpec.html#a3.2.2.1_Adding_Files_and_Archives_for_the_Job
>
> Thanks,
> Abhishek
>
> On Mon, Apr 25, 2016 at 2:21 AM, Serega Sheypak <se...@gmail.com>
> wrote:
>
> > Hi, oozie workflow by default expects special file layout:
> >
> > The directory structure looks like this:
> >
> > - wf-app-dir/workflow.xml
> > - wf-app-dir/lib
> > - wf-app-dir/lib/myJavaClasses.JAR
> >
> >
> > Is there any way to specify custom wf--app-dir/lib for each oozie
> workflow
> > action? My workflow actions are java-actions and I want to avoid jar-hell
> > in /lib dir common for the whole workflow.
> >
>
Re: Is there any way to specify custom lib dir for each workflow action
Posted by Abhishek Bafna <ba...@gmail.com>.
Hi,
Currently, there is not way specify the separate wf-app-dir/lib for
individual actions. But you can use the file and archive elements for each
java action.
https://oozie.apache.org/docs/4.2.0/WorkflowFunctionalSpec.html#a3.2.2.1_Adding_Files_and_Archives_for_the_Job
Thanks,
Abhishek
On Mon, Apr 25, 2016 at 2:21 AM, Serega Sheypak <se...@gmail.com>
wrote:
> Hi, oozie workflow by default expects special file layout:
>
> The directory structure looks like this:
>
> - wf-app-dir/workflow.xml
> - wf-app-dir/lib
> - wf-app-dir/lib/myJavaClasses.JAR
>
>
> Is there any way to specify custom wf--app-dir/lib for each oozie workflow
> action? My workflow actions are java-actions and I want to avoid jar-hell
> in /lib dir common for the whole workflow.
>