You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Ankit Jain <an...@gmail.com> on 2011/07/28 07:27:57 UTC
Error while indexing the LZO file
Hi all,
I tried to index the lzo file but got the following error while indexing the
lzo file :
java.lang.ClassCastException:
com.hadoop.compression.lzo.LzopCodec$LzopDecompressor cannot be cast to
com.hadoop.compression.lzo.LzopDecompressor
at
com.hadoop.mapreduce.LzoSplitRecordReader.initialize(LzoSplitRecordReader.java:52)
at
org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:418)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:620)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
I have performed following steps:-
1. $sudo apt-get install liblzo2-dev
2. Download the Hadoop-lzo clone from github repo (
https://github.com/kevinweil/hadoop-lzo )
3. Build the Hadoop-lzo project.
4. Copy the hadoop-lzo-*.jar file at $HADOOP_HOME/lib dir of cluster nodes
5. Copy the hadoop-lzo-install-dir/build/hadoop-lzo-*/native library at
$HADOOP_HOME/lib dir of cluster nodes.
6. core-site.xml:
<property>
<name>io.compression.codecs</name>
<value>org.apache.hadoop.io.compress.GzipCodec
,org.apache.hadoop.io.compress.DefaultCodec,
com.hadoop.compression.lzo.LzoCodec,
com.hadoop.compression.lzo.LzopCodec,
org.apache.hadoop.io.compress.BZip2Codec
</value>
</property>
<property>
<name>io.compression.codec.lzo.class</name>
<value>com.hadoop.compression.lzo.LzoCodec</value>
</property>
7. mapred-site.xml:
<property>
<name>mapred.child.env</name>
<value>JAVA_LIBRARY_PATH=/opt/ladap/common/hadoop-0.20.2/lib/native/Linux-i386-32/*</value>
</property>
<property>
<name>mapred.map.output.compression.codec</name>
<value>com.hadoop.compression.lzo.LzoCodec</value>
</property>
8. hadoop-env.sh
export
HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/home/ankit/hadoop-0.20.1/lib/hadoop-lzo-0.4.12.jar
export JAVA_LIBRARY_PATH=/home/ankit/hadoop-0.20.1/lib/native/Linux-i386-32/
9. Restart the cluster
10. uploaded lzo file into hdfs
11. Runned the following command for indexing:
bin/hadoop jar path/to/hadoop-lzo-*.jar
com.hadoop.compression.lzo.LzoIndexer lzofile.lzo