How to properly update oozie sharelibs

[Morgrim Murdargent <mo...@gmail.com>]

Hello !

I send you this mail because I would like to update the oozie sharelibs
(oozie 4.1) with additionnal libs and I am not sure of the proper way to do
it.

I created the sharelibs following this doc :
http://blog.cloudera.com/blog/2014/05/how-to-use-the-sharelib-in-apache-oozie-cdh-5/

I would like to add a new folder for hbase because there is no hbase folder
in my oozie sharelibs.

1. Is it normal that there is not the hbase libs by default in the tarball
for oozie sharelibs ?

2. How to properly add a lib folder for hbase and then update the sharelibs
in order to have a new lib_yyyymmddhhmiss folder ?

BR.

Morgrim.

Re: How to properly update oozie sharelibs

[Morgrim Murdargent <mo...@gmail.com>]

Hello Robert, Ahmed.

I'm not using Cloudera, I'm using HDP in fact.
But the solution you provide will work also for me.

Thank to both of you guys o/

Best regards.

Morgrim

On Tue, Apr 12, 2016 at 9:06 PM, Robert Kanter <rk...@cloudera.com> wrote:

> When you do "Install Sharelib" in Cloudera Manager, it simply runs the
> 'oozie-setup sharelib create -fs FS_URI [-locallib SHARED_LIBRARY]' command
> mentioned in
>
> http://blog.cloudera.com/blog/2014/05/how-to-use-the-sharelib-in-apache-oozie-cdh-5/
> That's also the recommended way to updating the sharelib.  Everytime you
> rerun it, it will create a new lib_<timestamp> dir.  If you want it to
> upload a custom sharelib dir, say "hbase", you can just add that dir in the
> correct place locally, and run that command.  The command is really just a
> glorified version of 'hadoop fs -put'.
>
> After you run that, you can either restart Oozie or run 'oozie admin
> -sharelibupdate' (also as described in the blog post) and Oozie will use
> the newest lib_<timestamp> dir.  If you directly add a dir or change the
> latest lib_<timestamp> dir in HDFS, you can also run this command or
> restart Oozie for it to notice the changes.
>
>
> - Robert
>
> On Sun, Apr 10, 2016 at 1:22 AM, Ahmed Kamal <ah...@badrit.com>
> wrote:
>
> > By the way as you are using CDH, you can install the sharelib again from
> > the cloudera manager oozie tab via "Install ShareLib" under Actions. It
> > will create a new folder with new directories. Also check this
> >
> >
> >
> http://capnjosh.com/blog/which-jar-files-do-you-need-in-the-sharelib-to-run-a-sqoop-job-via-oozie-that-imports-to-hbase/
> >
> > On Sun, Apr 10, 2016 at 10:16 AM, Ahmed Kamal <ah...@badrit.com>
> > wrote:
> >
> > > The sharelib has its path on the HDFS. If you are using Cloudera, this
> > > sharelib directory is created by the CDH. You will find it
> > > at /user/oozie/share/lib/. Each component has its own folder, user
> hadoop
> > > fs -put to upload all the jars you want to upload.
> > >
> > > On Mon, Apr 4, 2016 at 7:02 PM, Morgrim Murdargent <
> > > morgrim.oozie@gmail.com> wrote:
> > >
> > >> Hello !
> > >>
> > >> I send you this mail because I would like to update the oozie
> sharelibs
> > >> (oozie 4.1) with additionnal libs and I am not sure of the proper way
> to
> > >> do
> > >> it.
> > >>
> > >> I created the sharelibs following this doc :
> > >>
> > >>
> >
> http://blog.cloudera.com/blog/2014/05/how-to-use-the-sharelib-in-apache-oozie-cdh-5/
> > >>
> > >> I would like to add a new folder for hbase because there is no hbase
> > >> folder
> > >> in my oozie sharelibs.
> > >>
> > >> 1. Is it normal that there is not the hbase libs by default in the
> > tarball
> > >> for oozie sharelibs ?
> > >>
> > >> 2. How to properly add a lib folder for hbase and then update the
> > >> sharelibs
> > >> in order to have a new lib_yyyymmddhhmiss folder ?
> > >>
> > >> BR.
> > >>
> > >> Morgrim.
> > >>
> > >
> > >
> > >
> > > --
> > > JAK ,
> > >
> > > Ahmed Kamal
> > > Junior Big Data Engineer , BadrIT
> > >
> >
> >
> >
> > --
> > JAK ,
> >
> > Ahmed Kamal
> > Junior Big Data Engineer , BadrIT
> >
>

Re: How to properly update oozie sharelibs

[Robert Kanter <rk...@cloudera.com>]

When you do "Install Sharelib" in Cloudera Manager, it simply runs the
'oozie-setup sharelib create -fs FS_URI [-locallib SHARED_LIBRARY]' command
mentioned in
http://blog.cloudera.com/blog/2014/05/how-to-use-the-sharelib-in-apache-oozie-cdh-5/
That's also the recommended way to updating the sharelib.  Everytime you
rerun it, it will create a new lib_<timestamp> dir.  If you want it to
upload a custom sharelib dir, say "hbase", you can just add that dir in the
correct place locally, and run that command.  The command is really just a
glorified version of 'hadoop fs -put'.

After you run that, you can either restart Oozie or run 'oozie admin
-sharelibupdate' (also as described in the blog post) and Oozie will use
the newest lib_<timestamp> dir.  If you directly add a dir or change the
latest lib_<timestamp> dir in HDFS, you can also run this command or
restart Oozie for it to notice the changes.


- Robert

On Sun, Apr 10, 2016 at 1:22 AM, Ahmed Kamal <ah...@badrit.com> wrote:

> By the way as you are using CDH, you can install the sharelib again from
> the cloudera manager oozie tab via "Install ShareLib" under Actions. It
> will create a new folder with new directories. Also check this
>
>
> http://capnjosh.com/blog/which-jar-files-do-you-need-in-the-sharelib-to-run-a-sqoop-job-via-oozie-that-imports-to-hbase/
>
> On Sun, Apr 10, 2016 at 10:16 AM, Ahmed Kamal <ah...@badrit.com>
> wrote:
>
> > The sharelib has its path on the HDFS. If you are using Cloudera, this
> > sharelib directory is created by the CDH. You will find it
> > at /user/oozie/share/lib/. Each component has its own folder, user hadoop
> > fs -put to upload all the jars you want to upload.
> >
> > On Mon, Apr 4, 2016 at 7:02 PM, Morgrim Murdargent <
> > morgrim.oozie@gmail.com> wrote:
> >
> >> Hello !
> >>
> >> I send you this mail because I would like to update the oozie sharelibs
> >> (oozie 4.1) with additionnal libs and I am not sure of the proper way to
> >> do
> >> it.
> >>
> >> I created the sharelibs following this doc :
> >>
> >>
> http://blog.cloudera.com/blog/2014/05/how-to-use-the-sharelib-in-apache-oozie-cdh-5/
> >>
> >> I would like to add a new folder for hbase because there is no hbase
> >> folder
> >> in my oozie sharelibs.
> >>
> >> 1. Is it normal that there is not the hbase libs by default in the
> tarball
> >> for oozie sharelibs ?
> >>
> >> 2. How to properly add a lib folder for hbase and then update the
> >> sharelibs
> >> in order to have a new lib_yyyymmddhhmiss folder ?
> >>
> >> BR.
> >>
> >> Morgrim.
> >>
> >
> >
> >
> > --
> > JAK ,
> >
> > Ahmed Kamal
> > Junior Big Data Engineer , BadrIT
> >
>
>
>
> --
> JAK ,
>
> Ahmed Kamal
> Junior Big Data Engineer , BadrIT
>

Re: How to properly update oozie sharelibs

[Ahmed Kamal <ah...@badrit.com>]

By the way as you are using CDH, you can install the sharelib again from
the cloudera manager oozie tab via "Install ShareLib" under Actions. It
will create a new folder with new directories. Also check this

http://capnjosh.com/blog/which-jar-files-do-you-need-in-the-sharelib-to-run-a-sqoop-job-via-oozie-that-imports-to-hbase/

On Sun, Apr 10, 2016 at 10:16 AM, Ahmed Kamal <ah...@badrit.com>
wrote:

> The sharelib has its path on the HDFS. If you are using Cloudera, this
> sharelib directory is created by the CDH. You will find it
> at /user/oozie/share/lib/. Each component has its own folder, user hadoop
> fs -put to upload all the jars you want to upload.
>
> On Mon, Apr 4, 2016 at 7:02 PM, Morgrim Murdargent <
> morgrim.oozie@gmail.com> wrote:
>
>> Hello !
>>
>> I send you this mail because I would like to update the oozie sharelibs
>> (oozie 4.1) with additionnal libs and I am not sure of the proper way to
>> do
>> it.
>>
>> I created the sharelibs following this doc :
>>
>> http://blog.cloudera.com/blog/2014/05/how-to-use-the-sharelib-in-apache-oozie-cdh-5/
>>
>> I would like to add a new folder for hbase because there is no hbase
>> folder
>> in my oozie sharelibs.
>>
>> 1. Is it normal that there is not the hbase libs by default in the tarball
>> for oozie sharelibs ?
>>
>> 2. How to properly add a lib folder for hbase and then update the
>> sharelibs
>> in order to have a new lib_yyyymmddhhmiss folder ?
>>
>> BR.
>>
>> Morgrim.
>>
>
>
>
> --
> JAK ,
>
> Ahmed Kamal
> Junior Big Data Engineer , BadrIT
>



-- 
JAK ,

Ahmed Kamal
Junior Big Data Engineer , BadrIT

Re: How to properly update oozie sharelibs

[Ahmed Kamal <ah...@badrit.com>]

The sharelib has its path on the HDFS. If you are using Cloudera, this
sharelib directory is created by the CDH. You will find it
at /user/oozie/share/lib/. Each component has its own folder, user hadoop
fs -put to upload all the jars you want to upload.

On Mon, Apr 4, 2016 at 7:02 PM, Morgrim Murdargent <mo...@gmail.com>
wrote:

> Hello !
>
> I send you this mail because I would like to update the oozie sharelibs
> (oozie 4.1) with additionnal libs and I am not sure of the proper way to do
> it.
>
> I created the sharelibs following this doc :
>
> http://blog.cloudera.com/blog/2014/05/how-to-use-the-sharelib-in-apache-oozie-cdh-5/
>
> I would like to add a new folder for hbase because there is no hbase folder
> in my oozie sharelibs.
>
> 1. Is it normal that there is not the hbase libs by default in the tarball
> for oozie sharelibs ?
>
> 2. How to properly add a lib folder for hbase and then update the sharelibs
> in order to have a new lib_yyyymmddhhmiss folder ?
>
> BR.
>
> Morgrim.
>



-- 
JAK ,

Ahmed Kamal
Junior Big Data Engineer , BadrIT