You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by w00t w00t <w0...@yahoo.de> on 2013/08/08 15:10:30 UTC
Oozie and Lzo on Hortonworks 1.2 - ClassNotFoundException
Hello,
I ran Hortonworks' Sandbox 1.2 and used Oozie successfully to run Java MR, Pig and Hive jobs.
Then I installed Lzo support which now seems to break my Oozie.
I adjusted core-site.xml in the following way:
<property>
<name>io.compression.codecs</name> <value>org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,com.hadoop.compression.lzo.LzoCodec,com.hadoop.compression.lzo.LzopCodec,org.apache.hadoop.io.compress.BZip2Codec</value>
</property>
<property>
<name>io.compression.codec.lzo.class</name>
<value>com.hadoop.compression.lzo.LzoCodec</value>
</property>
I now have the same situation as documented here:
http://mapredit.blogspot.com/2013/02/lzo-compression-with-oozie.html
Oozie can't start any MR jobs and reports:
ClassNotFound error (java.lang.ClassNotFoundException: Class com.hadoop.compression.lzo.LzoCodec not found)
I am not sure where to place /usr/lib/hadoop/lib/hadoop-lzo-0.5.0.jar... I tried several things but was not successful..
I read that it needs to be put into /var/lib/oozie.
I am not sure whether this is the right location as there is the following directory strucuture:
/var/lib/oozie/oozie-server
/var/lib/oozie/oozie-server/conf
/var/lib/oozie/oozie-server/webapps
/var/lib/oozie/oozie-server/work
=> seems for me to be the Oozie Tomcat application.
Another possible location is /usr/lib/oozie/lib.
But again, putting the jar here and restarting Oozie did not solve the problem.
Can you help me to fix this problem?
I appreciate any help.
Re: Oozie and Lzo on Hortonworks 1.2 - ClassNotFoundException
Posted by w00t w00t <w0...@yahoo.de>.
Thank you very much for your replies.
I tried the sharelib configuration, but that didn't work... I have no clue why it is not working.
However, I got it working.
Here is a how-to from the Hortonworks documentation:
http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.2.1/bk_installing_manually_book/content/rpm-chap8-1.html
In this how-to, the library hadoop-lzo-0.5.0.jar gets packed into the war-file of Oozie
Thanks for all of your support!
________________________________
Von: Alejandro Abdelnur <tu...@gmail.com>
An: "user@oozie.apache.org" <us...@oozie.apache.org>
CC: "user@oozie.apache.org" <us...@oozie.apache.org>
Gesendet: 18:13 Donnerstag, 8.August 2013
Betreff: Re: Oozie and Lzo on Hortonworks 1.2 - ClassNotFoundException
i'm not familiar with the hw oozie setup, but if you put the lzo jar in the same dirs the oozie-core jar is, it should work.
if not i'd suggest reaching out the hw alias for their recommended way of doing it
thx
Alejandro
(phone typing)
On Aug 8, 2013, at 6:10, w00t w00t <w0...@yahoo.de> wrote:
> Hello,
> I ran Hortonworks' Sandbox 1.2 and used Oozie successfully to run Java MR, Pig and Hive jobs.
>
> Then I installed Lzo support which now seems to break my Oozie.
> I adjusted core-site.xml in the following way:
> <property>
> <name>io.compression.codecs</name> <value>org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,com.hadoop.compression.lzo.LzoCodec,com.hadoop.compression.lzo.LzopCodec,org.apache.hadoop.io.compress.BZip2Codec</value>
> </property>
> <property>
> <name>io.compression.codec.lzo.class</name>
> <value>com.hadoop.compression.lzo.LzoCodec</value>
> </property>
>
>
> I now have the same situation as documented here:
> http://mapredit.blogspot.com/2013/02/lzo-compression-with-oozie.html
>
> Oozie can't start any MR jobs and reports:
> ClassNotFound error (java.lang.ClassNotFoundException: Class com.hadoop.compression.lzo.LzoCodec not found)
>
> I am not sure where to place /usr/lib/hadoop/lib/hadoop-lzo-0.5.0.jar... I tried several things but was not successful..
>
> I read that it needs to be put into /var/lib/oozie.
> I am not sure whether this is the right location as there is the following directory strucuture:
> /var/lib/oozie/oozie-server
> /var/lib/oozie/oozie-server/conf
> /var/lib/oozie/oozie-server/webapps
> /var/lib/oozie/oozie-server/work
>
> => seems for me to be the Oozie Tomcat application.
>
> Another possible location is /usr/lib/oozie/lib.
> But again, putting the jar here and restarting Oozie did not solve the problem.
>
> Can you help me to fix this problem?
>
> I appreciate any help.
Re: Oozie and Lzo on Hortonworks 1.2 - ClassNotFoundException
Posted by Alejandro Abdelnur <tu...@gmail.com>.
i'm not familiar with the hw oozie setup, but if you put the lzo jar in the same dirs the oozie-core jar is, it should work.
if not i'd suggest reaching out the hw alias for their recommended way of doing it
thx
Alejandro
(phone typing)
On Aug 8, 2013, at 6:10, w00t w00t <w0...@yahoo.de> wrote:
> Hello,
> I ran Hortonworks' Sandbox 1.2 and used Oozie successfully to run Java MR, Pig and Hive jobs.
>
> Then I installed Lzo support which now seems to break my Oozie.
> I adjusted core-site.xml in the following way:
> <property>
> <name>io.compression.codecs</name> <value>org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,com.hadoop.compression.lzo.LzoCodec,com.hadoop.compression.lzo.LzopCodec,org.apache.hadoop.io.compress.BZip2Codec</value>
> </property>
> <property>
> <name>io.compression.codec.lzo.class</name>
> <value>com.hadoop.compression.lzo.LzoCodec</value>
> </property>
>
>
> I now have the same situation as documented here:
> http://mapredit.blogspot.com/2013/02/lzo-compression-with-oozie.html
>
> Oozie can't start any MR jobs and reports:
> ClassNotFound error (java.lang.ClassNotFoundException: Class com.hadoop.compression.lzo.LzoCodec not found)
>
> I am not sure where to place /usr/lib/hadoop/lib/hadoop-lzo-0.5.0.jar... I tried several things but was not successful..
>
> I read that it needs to be put into /var/lib/oozie.
> I am not sure whether this is the right location as there is the following directory strucuture:
> /var/lib/oozie/oozie-server
> /var/lib/oozie/oozie-server/conf
> /var/lib/oozie/oozie-server/webapps
> /var/lib/oozie/oozie-server/work
>
> => seems for me to be the Oozie Tomcat application.
>
> Another possible location is /usr/lib/oozie/lib.
> But again, putting the jar here and restarting Oozie did not solve the problem.
>
> Can you help me to fix this problem?
>
> I appreciate any help.
Re: Oozie and Lzo on Hortonworks 1.2 - ClassNotFoundException
Posted by Edward Capriolo <ed...@gmail.com>.
This may not be helpful but you are better off with snappy compression. Its
comparable speed/compression wise with lzo and does not have licensing
issues with hadoop
On Thu, Aug 8, 2013 at 9:10 AM, w00t w00t <w0...@yahoo.de> wrote:
> Hello,
> I ran Hortonworks' Sandbox 1.2 and used Oozie successfully to run Java MR,
> Pig and Hive jobs.
>
> Then I installed Lzo support which now seems to break my Oozie.
> I adjusted core-site.xml in the following way:
> <property>
> <name>io.compression.codecs</name>
> <value>org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,com.hadoop.compression.lzo.LzoCodec,com.hadoop.compression.lzo.LzopCodec,org.apache.hadoop.io.compress.BZip2Codec</value>
> </property>
> <property>
> <name>io.compression.codec.lzo.class</name>
> <value>com.hadoop.compression.lzo.LzoCodec</value>
> </property>
>
>
> I now have the same situation as documented here:
> http://mapredit.blogspot.com/2013/02/lzo-compression-with-oozie.html
>
> Oozie can't start any MR jobs and reports:
> ClassNotFound error (java.lang.ClassNotFoundException: Class
> com.hadoop.compression.lzo.LzoCodec not found)
>
> I am not sure where to place /usr/lib/hadoop/lib/hadoop-lzo-0.5.0.jar... I
> tried several things but was not successful..
>
> I read that it needs to be put into /var/lib/oozie.
> I am not sure whether this is the right location as there is the following
> directory strucuture:
> /var/lib/oozie/oozie-server
> /var/lib/oozie/oozie-server/conf
> /var/lib/oozie/oozie-server/webapps
> /var/lib/oozie/oozie-server/work
>
> => seems for me to be the Oozie Tomcat application.
>
> Another possible location is /usr/lib/oozie/lib.
> But again, putting the jar here and restarting Oozie did not solve the
> problem.
>
> Can you help me to fix this problem?
>
> I appreciate any help.
Re: Oozie and Lzo on Hortonworks 1.2 - ClassNotFoundException
Posted by Ken Ernst <ke...@mobileposse.com>.
This article may helpŠ
http://blog.cloudera.com/blog/2012/12/how-to-use-the-sharelib-in-apache-ooz
ie/
On 8/8/13 9:10 AM, "w00t w00t" <w0...@yahoo.de> wrote:
>Hello,
>I ran Hortonworks' Sandbox 1.2 and used Oozie successfully to run Java
>MR, Pig and Hive jobs.
>
>Then I installed Lzo support which now seems to break my Oozie.
>I adjusted core-site.xml in the following way:
> <property>
> <name>io.compression.codecs</name>
><value>org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compre
>ss.DefaultCodec,com.hadoop.compression.lzo.LzoCodec,com.hadoop.compression
>.lzo.LzopCodec,org.apache.hadoop.io.compress.BZip2Codec</value>
> </property>
> <property>
> <name>io.compression.codec.lzo.class</name>
> <value>com.hadoop.compression.lzo.LzoCodec</value>
> </property>
>
>
>I now have the same situation as documented here:
>http://mapredit.blogspot.com/2013/02/lzo-compression-with-oozie.html
>
>Oozie can't start any MR jobs and reports:
>ClassNotFound error (java.lang.ClassNotFoundException: Class
>com.hadoop.compression.lzo.LzoCodec not found)
>
>I am not sure where to place /usr/lib/hadoop/lib/hadoop-lzo-0.5.0.jar...
>I tried several things but was not successful..
>
>I read that it needs to be put into /var/lib/oozie.
>I am not sure whether this is the right location as there is the
>following directory strucuture:
>/var/lib/oozie/oozie-server
>/var/lib/oozie/oozie-server/conf
>/var/lib/oozie/oozie-server/webapps
>/var/lib/oozie/oozie-server/work
>
>=> seems for me to be the Oozie Tomcat application.
>
>Another possible location is /usr/lib/oozie/lib.
>But again, putting the jar here and restarting Oozie did not solve the
>problem.
>
>Can you help me to fix this problem?
>
>I appreciate any help.