You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Alex Luya <al...@gmail.com> on 2010/08/17 15:44:03 UTC
lzo library existed in classpath,Why can't it be refered by job?
Hello
I followed this guide:http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201006.mbox/<AANLkTileo-
q8USEiP8Y3Na9pDYHlyUFIPpR0In0LkpJm@mail.gmail.com> ,then run:
hadoop jar hadoop-examples-0.20.2+320.jar grep input output 'dfs[a-z.]+',
got error:"Caused by: java.lang.ClassNotFoundException: com.hadoop.compression.lzo.LzoCodec",but I run:
ps -eaf|grep tasktracker|grep classpath,got
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
alex 1743 1 0 19:31 ? 00:00:04 /usr/local/hadoop/jdk1.6.0_21/bin/java -Xmx1000m -
Dhadoop.log.dir=/usr/local/hadoop/hadoop-0.20.2/bin/../logs -Dhadoop.log.file=hadoop-alex-tasktracker-Hadoop-01.log -
Dhadoop.home.dir=/usr/local/hadoop/hadoop-0.20.2/bin/.. -Dhadoop.id.str=alex -Dhadoop.root.logger=INFO,DRFA -
Djava.library.path=/usr/local/hadoop/hadoop-0.20.2/lib/native/Linux-amd64-64:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/native/Linux-amd64-64 -
Dhadoop.policy.file=hadoop-policy.xml -classpath
/usr/local/hadoop/hadoop-0.20.2/conf:/usr/local/hadoop/jdk1.6.0_21/lib/tools.jar:/usr/local/hadoop/hadoop-0.20.2/bin/..:/usr/local/hadoop/hadoop-0.20.2/bin/../hadoop-
core-0.20.2+320.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/commons-cli-1.2.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/commons-
codec-1.3.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/commons-el-1.0.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/commons-
httpclient-3.0.1.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/commons-logging-1.0.4.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/commons-logging-
api-1.0.4.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/commons-
net-1.4.1.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/core-3.1.1.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/hadoop-
fairscheduler-0.20.2+320.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/hadoop-lzo-0.4.4.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/hadoop-scribe-
log4j-0.20.2+320.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/hsqldb.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/jackson-
core-asl-1.0.1.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/jackson-mapper-asl-1.0.1.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/jasper-
compiler-5.5.12.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/jasper-
runtime-5.5.12.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/jets3t-0.6.1.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/jetty-6.1.14.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/jetty-
util-6.1.14.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/junit-4.5.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/kfs-0.2.2.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/libfb303.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/libthrift.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/log4j-1.2.15.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/mockito-
all-1.8.2.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/mysql-connector-java-5.0.8-
bin.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/oro-2.0.8.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/servlet-
api-2.5-6.1.14.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/slf4j-api-1.4.3.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/slf4j-
log4j12-1.4.3.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/xmlenc-0.52.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/jsp-2.1/jsp-2.1.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/jsp-2.1/jsp-
api-2.1.jar:/usr/local/hadoop/hadoop-0.20.2/lib/hadoop-lzo-0.4.4.jar org.apache.hadoop.mapred.TaskTracker
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
See,lzo library appeared in classpath,so questions are:
1,Here,lzo library appeared tow times,so if lzo library has been put in the $HADOOP_HOME/lib and native libraries have been put in
$HADOOP_HOME/lib/native,whether following settings isn't needed:
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
hadoop-env.sh
export HADOOP_CLASSPATH=/usr/lib/hadoop/lib/hadoop-lzo-0.4.3.jar
export JAVA_LIBRARY_PATH=/usr/lib/hadoop/lib/native/Linux-i386-32 (or
the 64bit version)
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
2,If lzo library appeared in the classpath,as above shows,whether this means that lzo library has been loaded
successfully,and if it was,why I get got error:"Caused by: java.lang.ClassNotFoundException: com.hadoop.compression.lzo.LzoCodec" .