You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Alex Luya <al...@gmail.com> on 2010/08/15 14:58:37 UTC
how to get lzo library library loaded?(error :Caused by: java.lang.ClassNotFoundException: com.hadoop.compression.lzo.LzoCodec)
Hi,
At every beginning,I run:hadoop jar hadoop-*-examples.jar grep input output 'dfs[a-z.]+' successfully,but when run:
nutch crawl url -dir crawl -depth 3,got errors:
------------------------------------------------------------------------- -------------------------------------------------------------------------
10/08/07 22:53:30 INFO crawl.Crawl: crawl started in: crawl
.....................................................................
10/08/07 22:53:30 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
Exception in thread "main" java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
.....................................................................
at org.apache.nutch.crawl.Crawl.main(Crawl.java:124)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
.....................................................................
... 9 more
Caused by: java.lang.IllegalArgumentException: Compression codec
org.apache.hadoop.io.compress.GzipCodec not found.
at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:96)
.....................................................................
... 14 more
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.io.compress.GzipCodec
.....................................................................
at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:89)
... 16 more
------------------------------------------------------------------------- -------------------------------------------------------------------------
So,here GzipCode didn't get loaded successfully,or maybe it will not be loaded by default,I don't know,but I think it should be,then I followed this
link:http://code.google.com/p/hadoop-gpl-compression/wiki/FAQ to install lzo and run:
"nutch crawl url -dir crawl -depth 3" again,got errors:
------------------------------------------------------------------------- -------------------------------------------------------------------------
10/08/07 22:40:41 INFO crawl.Crawl: crawl started in: crawl
.....................................................................
10/08/07 22:40:42 INFO crawl.Injector: Injector: Converting injected urls to crawl db entries.
10/08/07 22:40:42 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
Exception in thread "main" java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
.....................................................................
at org.apache.nutch.crawl.Injector.inject(Injector.java:211)
at org.apache.nutch.crawl.Crawl.main(Crawl.java:124)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
.....................................................................
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
... 9 more
Caused by: java.lang.IllegalArgumentException: Compression codec
org.apache.hadoop.io.compress.GzipCodec not found.
.....................................................................
at org.apache.hadoop.mapred.TextInputFormat.configure(TextInputFormat.java:41)
... 14 more
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.io.compress.GzipCodec
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
.....................................................................
at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:89)
... 16 more
------------------------------------------------------------------------- -------------------------------------------------------------------------
run:hadoop jar hadoop-*-examples.jar grep input output 'dfs[a-z.]+',got errors:
------------------------------------------------------------------------- -------------------------------------------------------------------------
java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at org.apache.hadoop.mapred.JobConf.getInputFormat(JobConf.java:400)
.....................................................................
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
.....................................................................
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
... 22 more
Caused by: java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lzo.LzoCodec
not found.
at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:96)
at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:134)
at org.apache.hadoop.mapred.TextInputFormat.configure(TextInputFormat.java:41)
... 27 more
Caused by: java.lang.ClassNotFoundException: com.hadoop.compression.lzo.LzoCodec
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
.....................................................................
at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:89)
... 29 more
------------------------------------------------------------------------- -------------------------------------------------------------------------
when I run " ps -aef|grep gpl",got output:
------------------------------------------------------------------------- -------------------------------------------------------------------------
alex 2267 1 1 22:04 pts/1 00:00:04 /usr/local/hadoop/jdk1.6.0_21/bin/java -Xmx200m -Dcom.sun.management.jmxremote
-..............................................
/usr/local/hadoop/hadoop-0.20.2/bin/../conf:/usr/local/hadoop/jdk1.6.0_21/lib/tools.jar:/usr/local/hadoop/hadoop-0.20.2/bin/..:/usr/local/hadoop/hadoop-0.20.2/bin/../hadoop-0.20.2-
core.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/commons-cli-1.2.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/commons-
codec-1.3.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/commons-el-1.0.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/commons-.-
net-1.4.1.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/core-3.1.1.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/hadoop-gpl-compression-0.2.0-
dev.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/jasper-
compiler-5.5.12.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/jasper-
runtime-5.5.12.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/jets3t-0.6.1.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/jetty-6.1.14.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/jetty-
.......................................-
log4j12-1.4.3.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/xmlenc-0.52.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/jsp-2.1/jsp-2.1.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/jsp-2.1/jsp-
api-2.1.jar org.apache.hadoop.hdfs.server.namenode.NameNode
See,two jars(hadoop-core and pgl) are existing in classpath,but seems that they can't be referenced by job,before this, I have tried to install
hadoop-lzo(http://github.com/kevinweil/hadoop-lzo),same errors,maybe hadoop-lzo only works for hadoop 0.20,not for 0.20.1/2,I don't know.After one
month,I haven't solved this problem,it's killing me,here I post all configure files,would you please help me dig problem out?Thank you.
core-site.xml
------------------------------------------------------------------------- -------------------------------------------------------------------------
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://AlexLuya:8020</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/alex/tmp</value>
</property>
<property>
<name>io.compression.codecs</name>
<value>org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.BZip2Codec,com.hadoop.compression.lzo.LzoCodec
</value>
</property>
<property>
<name>io.compression.codec.lzo.class</name>
<value>com.hadoop.compression.lzo.LzoCodec</value>
</property>
</configuration>
------------------------------------------------------------------------- -------------------------------------------------------------------------
mapreduce.xml
------------------------------------------------------------------------- -------------------------------------------------------------------------
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>AlexLuya:9001</value>
</property>
<property>
<name>mapred.tasktracker.reduce.tasks.maximum</name>
<value>1</value>
</property>
<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<value>1</value>
</property>
<property>
<name>mapred.local.dir</name>
<value>/home/alex/hadoop/mapred/local</value>
</property>
<property>
<name>mapred.system.dir</name>
<value>/tmp/hadoop/mapred/system</value>
</property>
<property>
<name>mapreduce.map.output.compress</name>
<value>true</value>
</property>
<property>
<name>mapreduce.map.output.compress.codec</name>
<value>com.hadoop.compression.lzo.LzoCodec</value>
</property>
</configuration>
------------------------------------------------------------------------- -------------------------------------------------------------------------
hadoop-env.sh
------------------------------------------------------------------------- -------------------------------------------------------------------------
# Set Hadoop-specific environment variables here.
# The only required environment variable is JAVA_HOME. All others are
# optional. When running a distributed configuration it is best to
# set JAVA_HOME in this file, so that it is correctly defined on
# remote nodes.
# The java implementation to use. Required.
export JAVA_HOME=/usr/local/hadoop/jdk1.6.0_21
# Extra Java CLASSPATH elements. Optional.
# export HADOOP_CLASSPATH=
# The maximum amount of heap to use, in MB. Default is 1000.
export HADOOP_HEAPSIZE=200
# Extra Java runtime options. Empty by default.
#export HADOOP_OPTS=-server
# Command specific options appended to HADOOP_OPTS when specified
export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_NAMENODE_OPTS"
export HADOOP_SECONDARYNAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_SECONDARYNAMENODE_OPTS"
export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_DATANODE_OPTS"
export HADOOP_BALANCER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_BALANCER_OPTS"
export HADOOP_JOBTRACKER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_JOBTRACKER_OPTS"
# export HADOOP_TASKTRACKER_OPTS=
# The following applies to multiple commands (fs, dfs, fsck, distcp etc)
# export HADOOP_CLIENT_OPTS
# Extra ssh options. Empty by default.
# export HADOOP_SSH_OPTS="-o ConnectTimeout=1 -o SendEnv=HADOOP_CONF_DIR"
# Where log files are stored. $HADOOP_HOME/logs by default.
# export HADOOP_LOG_DIR=${HADOOP_HOME}/logs
# File naming remote slave hosts. $HADOOP_HOME/conf/slaves by default.
# export HADOOP_SLAVES=${HADOOP_HOME}/conf/slaves
# host:path where hadoop code should be rsync'd from. Unset by default.
# export HADOOP_MASTER=master:/home/$USER/src/hadoop
# Seconds to sleep between slave commands. Unset by default. This
# can be useful in large clusters, where, e.g., slave rsyncs can
# otherwise arrive faster than the master can service them.
# export HADOOP_SLAVE_SLEEP=0.1
# The directory where pid files are stored. /tmp by default.
# export HADOOP_PID_DIR=/var/hadoop/pids
# A string representing this instance of hadoop. $USER by default.
#export HADOOP_IDENT_STRING=$USER
# The scheduling priority for daemon processes. See 'man nice'.
# export HADOOP_NICENESS=10
------------------------------------------------------------------------- -------------------------------------------------------------------------
Re: how to get lzo library library loaded?(error :Caused by: java.lang.ClassNotFoundException: com.hadoop.compression.lzo.LzoCodec)
Posted by Alex Luya <al...@gmail.com>.
Hello
I use java.I think the problem is that I can't get lzo library loaded,and I can't get example run successfully.so It is not the problem of
programing language.
On Monday, August 16, 2010 03:43:12 pm rosefinny111 wrote:
> hi friend u write program which language
> means it in the java and php and etc. tell the answer to next post to me.
>
> <configuration>
> <property>
> <name>mapred.job.tracker</name>
> <value>AlexLuya:9001</value>
> </property>
> <property>
> <name>mapred.tasktracker.reduce.tasks.maximum</name>
> <value>1</value>
> </property>
> <property>
> <name>mapred.tasktracker.map.tasks.maximum</name>
> <value>1</value>
> </property>
> hi friend u write program which language
> means it in the java and php and etc. tell the answer to next post to me.
> <property>
> <name>mapred.local.dir</name>
> <value>/home/alex/hadoop/mapred/local</value>
> </property>
> <property>
> <name>mapred.system.dir</name>
> <value>/tmp/hadoop/mapred/system</value>
> </property>
> <property>
> <name>mapreduce.map.output.compress</name>
> <value>true</value>
> </property>
> <property>
> <name>mapreduce.map.output.compress.codec</name>
> <value>com.hadoop.compression.lzo.LzoCodec</value>
> </property>
> </configuration>
>
> regards,
> phe9oxis,
> http://www.guidebuddha.com
Re: how to get lzo library library loaded?(error :Caused by:
java.lang.ClassNotFoundException: com.hadoop.compression.lzo.LzoCodec)
Posted by rosefinny111 <vl...@gmail.com>.
hi friend u write program which language
means it in the java and php and etc. tell the answer to next post to me.
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>AlexLuya:9001</value>
</property>
<property>
<name>mapred.tasktracker.reduce.tasks.maximum</name>
<value>1</value>
</property>
<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<value>1</value>
</property>
hi friend u write program which language
means it in the java and php and etc. tell the answer to next post to me.
<property>
<name>mapred.local.dir</name>
<value>/home/alex/hadoop/mapred/local</value>
</property>
<property>
<name>mapred.system.dir</name>
<value>/tmp/hadoop/mapred/system</value>
</property>
<property>
<name>mapreduce.map.output.compress</name>
<value>true</value>
</property>
<property>
<name>mapreduce.map.output.compress.codec</name>
<value>com.hadoop.compression.lzo.LzoCodec</value>
</property>
</configuration>
regards,
phe9oxis,
http://www.guidebuddha.com
--
View this message in context: http://lucene.472066.n3.nabble.com/how-to-get-lzo-library-library-loaded-error-Caused-by-java-lang-ClassNotFoundException-com-hadoop-co-tp1157220p1166613.html
Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
Re: how to get lzo library library loaded?(error :Caused by: java.lang.ClassNotFoundException: com.hadoop.compression.lzo.LzoCodec)
Posted by Alex Luya <al...@gmail.com>.
Thank you,
I followed your guide:http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201006.mbox/<AANLkTileo-
q8USEiP8Y3Na9pDYHlyUFIPpR0In0LkpJm@mail.gmail.com> ,same error as before,when I run:
ps -eaf|grep tasktracker|grep classpath
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
alex 1743 1 0 19:31 ? 00:00:04 /usr/local/hadoop/jdk1.6.0_21/bin/java -Xmx1000m -
Dhadoop.log.dir=/usr/local/hadoop/hadoop-0.20.2/bin/../logs -Dhadoop.log.file=hadoop-alex-tasktracker-Hadoop-01.log -
Dhadoop.home.dir=/usr/local/hadoop/hadoop-0.20.2/bin/.. -Dhadoop.id.str=alex -Dhadoop.root.logger=INFO,DRFA -
Djava.library.path=/usr/local/hadoop/hadoop-0.20.2/lib/native/Linux-amd64-64:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/native/Linux-amd64-64 -
Dhadoop.policy.file=hadoop-policy.xml -classpath
/usr/local/hadoop/hadoop-0.20.2/conf:/usr/local/hadoop/jdk1.6.0_21/lib/tools.jar:/usr/local/hadoop/hadoop-0.20.2/bin/..:/usr/local/hadoop/hadoop-0.20.2/bin/../hadoop-
core-0.20.2+320.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/commons-cli-1.2.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/commons-
codec-1.3.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/commons-el-1.0.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/commons-
httpclient-3.0.1.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/commons-logging-1.0.4.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/commons-logging-
api-1.0.4.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/commons-
net-1.4.1.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/core-3.1.1.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/hadoop-
fairscheduler-0.20.2+320.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/hadoop-lzo-0.4.4.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/hadoop-scribe-
log4j-0.20.2+320.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/hsqldb.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/jackson-
core-asl-1.0.1.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/jackson-mapper-asl-1.0.1.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/jasper-
compiler-5.5.12.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/jasper-
runtime-5.5.12.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/jets3t-0.6.1.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/jetty-6.1.14.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/jetty-
util-6.1.14.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/junit-4.5.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/kfs-0.2.2.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/libfb303.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/libthrift.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/log4j-1.2.15.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/mockito-
all-1.8.2.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/mysql-connector-java-5.0.8-
bin.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/oro-2.0.8.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/servlet-
api-2.5-6.1.14.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/slf4j-api-1.4.3.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/slf4j-
log4j12-1.4.3.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/xmlenc-0.52.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/jsp-2.1/jsp-2.1.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/jsp-2.1/jsp-
api-2.1.jar:/usr/local/hadoop/hadoop-0.20.2/lib/hadoop-lzo-0.4.4.jar org.apache.hadoop.mapred.TaskTracker
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
See,lzo library appeared in classpath,so questions are:
1,Here,lzo library appeared tow times,so if lzo library has been put in the $HADOOP_HOME/lib and native libraries have been put in
$HADOOP_HOME/lib/native,whether following settings isn't needed:
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
hadoop-env.sh
export HADOOP_CLASSPATH=/usr/lib/hadoop/lib/hadoop-lzo-0.4.3.jar
export JAVA_LIBRARY_PATH=/usr/lib/hadoop/lib/native/Linux-i386-32 (or
the 64bit version)
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
2,the basic problem is that if lzo library appeared in the classpath,as above shows,whether this means that lzo library has been loaded
successfully,and if it was,why I get got error:"Caused by: java.lang.ClassNotFoundException: com.hadoop.compression.lzo.LzoCodec" when run:
hadoop jar hadoop-examples-0.20.2+320.jar grep input output 'dfs[a-z.]+',
On Monday, August 16, 2010 06:50:28 am Josh Patterson wrote:
> Alex,
> LZO can be a pain, we've all seen it; I have a few tips I've compiled
> that might help you (I've posted these before):
>
> http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201006.mbox/%3C
> AANLkTileo-q8USEiP8Y3Na9pDYHlyUFIPpR0In0LkpJm@mail.gmail.com%3E
>
> walk through some of those and see if that does help you. I've hung
> LZO up before and what has worked for me was to step through each
> phase, re-doing the step as needed until I've got it working again.
>
> Josh Patterson
> Cloudera
>
> On Sun, Aug 15, 2010 at 7:58 AM, Alex Luya <al...@gmail.com> wrote:
> > Hi,
> >
> > At every beginning,I run:hadoop jar hadoop-*-examples.jar grep input
> > output 'dfs[a-z.]+' successfully,but when run:
> >
> > nutch crawl url -dir crawl -depth 3,got errors:
> >
> > -------------------------------------------------------------------------
> > -------------------------------------------------------------------------
> >
> > 10/08/07 22:53:30 INFO crawl.Crawl: crawl started in: crawl
> >
> > .....................................................................
> >
> > 10/08/07 22:53:30 WARN mapred.JobClient: Use GenericOptionsParser for
> > parsing the arguments. Applications should implement Tool for the same.
> >
> > Exception in thread "main" java.lang.RuntimeException: Error in
> > configuring object
> >
> > at
> > org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93
> > )
> >
> > at
> > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
> >
> > .....................................................................
> >
> > at org.apache.nutch.crawl.Crawl.main(Crawl.java:124)
> >
> > Caused by: java.lang.reflect.InvocationTargetException
> >
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >
> > at
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java
> > :39)
> >
> > .....................................................................
> >
> > ... 9 more
> >
> > Caused by: java.lang.IllegalArgumentException: Compression codec
> >
> > org.apache.hadoop.io.compress.GzipCodec not found.
> >
> > at
> > org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(Com
> > pressionCodecFactory.java:96)
> >
> > .....................................................................
> >
> > ... 14 more
> >
> > Caused by: java.lang.ClassNotFoundException:
> > org.apache.hadoop.io.compress.GzipCodec
> >
> > .....................................................................
> >
> > at
> > org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(Com
> > pressionCodecFactory.java:89)
> >
> > ... 16 more
> >
> > -------------------------------------------------------------------------
> > -------------------------------------------------------------------------
> >
> > So,here GzipCode didn't get loaded successfully,or maybe it will not be
> > loaded by default,I don't know,but I think it should be,then I followed
> > this
> >
> > link:http://code.google.com/p/hadoop-gpl-compression/wiki/FAQ to install
> > lzo and run:
> >
> > "nutch crawl url -dir crawl -depth 3" again,got errors:
> >
> > -------------------------------------------------------------------------
> > -------------------------------------------------------------------------
> >
> > 10/08/07 22:40:41 INFO crawl.Crawl: crawl started in: crawl
> >
> > .....................................................................
> >
> > 10/08/07 22:40:42 INFO crawl.Injector: Injector: Converting injected urls
> > to crawl db entries.
> >
> > 10/08/07 22:40:42 WARN mapred.JobClient: Use GenericOptionsParser for
> > parsing the arguments. Applications should implement Tool for the same.
> >
> > Exception in thread "main" java.lang.RuntimeException: Error in
> > configuring object
> >
> > at
> > org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93
> > )
> >
> > .....................................................................
> >
> > at org.apache.nutch.crawl.Injector.inject(Injector.java:211)
> >
> > at org.apache.nutch.crawl.Crawl.main(Crawl.java:124)
> >
> > Caused by: java.lang.reflect.InvocationTargetException
> >
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >
> > .....................................................................
> >
> > at
> > org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88
> > )
> >
> > ... 9 more
> >
> > Caused by: java.lang.IllegalArgumentException: Compression codec
> >
> > org.apache.hadoop.io.compress.GzipCodec not found.
> >
> > .....................................................................
> >
> > at
> > org.apache.hadoop.mapred.TextInputFormat.configure(TextInputFormat.java:4
> > 1)
> >
> > ... 14 more
> >
> > Caused by: java.lang.ClassNotFoundException:
> > org.apache.hadoop.io.compress.GzipCodec
> >
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> >
> > at java.security.AccessController.doPrivileged(Native Method)
> >
> > .....................................................................
> >
> > at
> > org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(Com
> > pressionCodecFactory.java:89)
> >
> > ... 16 more
> >
> > -------------------------------------------------------------------------
> > -------------------------------------------------------------------------
> >
> > run:hadoop jar hadoop-*-examples.jar grep input output 'dfs[a-z.]+',got
> > errors:
> >
> > -------------------------------------------------------------------------
> > -------------------------------------------------------------------------
> >
> > java.lang.RuntimeException: Error in configuring object
> >
> > at
> > org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93
> > )
> >
> > at
> > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
> >
> > at
> > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:1
> > 17)
> >
> > at org.apache.hadoop.mapred.JobConf.getInputFormat(JobConf.java:400)
> >
> > .....................................................................
> >
> > at java.lang.reflect.Method.invoke(Method.java:597)
> >
> > at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> >
> > Caused by: java.lang.reflect.InvocationTargetException
> >
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >
> > .....................................................................
> >
> > at
> > org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88
> > )
> >
> > ... 22 more
> >
> > Caused by: java.lang.IllegalArgumentException: Compression codec
> > com.hadoop.compression.lzo.LzoCodec
> >
> > not found.
> >
> > at
> > org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(Com
> > pressionCodecFactory.java:96)
> >
> > at
> > org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionC
> > odecFactory.java:134)
> >
> > at
> > org.apache.hadoop.mapred.TextInputFormat.configure(TextInputFormat.java:4
> > 1)
> >
> > ... 27 more
> >
> > Caused by: java.lang.ClassNotFoundException:
> > com.hadoop.compression.lzo.LzoCodec
> >
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> >
> > at java.security.AccessController.doPrivileged(Native Method)
> >
> > .....................................................................
> >
> > at
> > org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(Com
> > pressionCodecFactory.java:89)
> >
> > ... 29 more
> >
> > -------------------------------------------------------------------------
> > -------------------------------------------------------------------------
> >
> > when I run " ps -aef|grep gpl",got output:
> >
> > -------------------------------------------------------------------------
> > -------------------------------------------------------------------------
> >
> > alex 2267 1 1 22:04 pts/1 00:00:04 /usr/local/hadoop/jdk1.6.0_21/bin/java
> > -Xmx200m -Dcom.sun.management.jmxremote
> >
> > -..............................................
> >
> > /usr/local/hadoop/hadoop-0.20.2/bin/../conf:/usr/local/hadoop/jdk1.6.0_21
> > /lib/tools.jar:/usr/local/hadoop/hadoop-0.20.2/bin/..:/usr/local/hadoop/h
> > adoop-0.20.2/bin/../hadoop-0.20.2-
> >
> > core.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/commons-cli-1.2.jar:/
> > usr/local/hadoop/hadoop-0.20.2/bin/../lib/commons-
> >
> > codec-1.3.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/commons-el-1.0.j
> > ar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/commons-.-
> >
> > net-1.4.1.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/core-3.1.1.jar:/
> > usr/local/hadoop/hadoop-0.20.2/bin/../lib/hadoop-gpl-compression-0.2.0-
> >
> > dev.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/hsqldb-1.8.0.10.jar:/u
> > sr/local/hadoop/hadoop-0.20.2/bin/../lib/jasper-
> >
> > compiler-5.5.12.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/jasper-
> >
> > runtime-5.5.12.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/jets3t-0.6.
> > 1.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/jetty-6.1.14.jar:/usr/lo
> > cal/hadoop/hadoop-0.20.2/bin/../lib/jetty-
> >
> > .......................................-
> >
> > log4j12-1.4.3.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/xmlenc-0.52.
> > jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/jsp-2.1/jsp-2.1.jar:/usr/l
> > ocal/hadoop/hadoop-0.20.2/bin/../lib/jsp-2.1/jsp-
> >
> > api-2.1.jar org.apache.hadoop.hdfs.server.namenode.NameNode
> >
> > See,two jars(hadoop-core and pgl) are existing in classpath,but seems
> > that they can't be referenced by job,before this, I have tried to
> > install hadoop-lzo(http://github.com/kevinweil/hadoop-lzo),same
> > errors,maybe hadoop-lzo only works for hadoop 0.20,not for 0.20.1/2,I
> > don't know.After one month,I haven't solved this problem,it's killing
> > me,here I post all configure files,would you please help me dig problem
> > out?Thank you.
> >
> > core-site.xml
> >
> > -------------------------------------------------------------------------
> > -------------------------------------------------------------------------
> >
> > <configuration>
> >
> > <property>
> >
> > <name>fs.default.name</name>
> >
> > <value>hdfs://AlexLuya:8020</value>
> >
> > </property>
> >
> > <property>
> >
> > <name>hadoop.tmp.dir</name>
> >
> > <value>/home/alex/tmp</value>
> >
> > </property>
> >
> > <property>
> >
> > <name>io.compression.codecs</name>
> >
> > <value>org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compr
> > ess.DefaultCodec,org.apache.hadoop.io.compress.BZip2Codec,com.hadoop.comp
> > ression.lzo.LzoCodec
> >
> > </value>
> >
> > </property>
> >
> > <property>
> >
> > <name>io.compression.codec.lzo.class</name>
> >
> > <value>com.hadoop.compression.lzo.LzoCodec</value>
> >
> > </property>
> >
> > </configuration>
> >
> > -------------------------------------------------------------------------
> > -------------------------------------------------------------------------
> >
> > mapreduce.xml
> >
> > -------------------------------------------------------------------------
> > -------------------------------------------------------------------------
> >
> > <configuration>
> >
> > <property>
> >
> > <name>mapred.job.tracker</name>
> >
> > <value>AlexLuya:9001</value>
> >
> > </property>
> >
> > <property>
> >
> > <name>mapred.tasktracker.reduce.tasks.maximum</name>
> >
> > <value>1</value>
> >
> > </property>
> >
> > <property>
> >
> > <name>mapred.tasktracker.map.tasks.maximum</name>
> >
> > <value>1</value>
> >
> > </property>
> >
> > <property>
> >
> > <name>mapred.local.dir</name>
> >
> > <value>/home/alex/hadoop/mapred/local</value>
> >
> > </property>
> >
> > <property>
> >
> > <name>mapred.system.dir</name>
> >
> > <value>/tmp/hadoop/mapred/system</value>
> >
> > </property>
> >
> > <property>
> >
> > <name>mapreduce.map.output.compress</name>
> >
> > <value>true</value>
> >
> > </property>
> >
> > <property>
> >
> > <name>mapreduce.map.output.compress.codec</name>
> >
> > <value>com.hadoop.compression.lzo.LzoCodec</value>
> >
> > </property>
> >
> > </configuration>
> >
> > -------------------------------------------------------------------------
> > -------------------------------------------------------------------------
> >
> > hadoop-env.sh
> >
> > -------------------------------------------------------------------------
> > -------------------------------------------------------------------------
> >
> > # Set Hadoop-specific environment variables here.
> >
> > # The only required environment variable is JAVA_HOME. All others are
> >
> > # optional. When running a distributed configuration it is best to
> >
> > # set JAVA_HOME in this file, so that it is correctly defined on
> >
> > # remote nodes.
> >
> > # The java implementation to use. Required.
> >
> > export JAVA_HOME=/usr/local/hadoop/jdk1.6.0_21
> >
> > # Extra Java CLASSPATH elements. Optional.
> >
> > # export HADOOP_CLASSPATH=
> >
> > # The maximum amount of heap to use, in MB. Default is 1000.
> >
> > export HADOOP_HEAPSIZE=200
> >
> > # Extra Java runtime options. Empty by default.
> >
> > #export HADOOP_OPTS=-server
> >
> > # Command specific options appended to HADOOP_OPTS when specified
> >
> > export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote
> > $HADOOP_NAMENODE_OPTS"
> >
> > export HADOOP_SECONDARYNAMENODE_OPTS="-Dcom.sun.management.jmxremote
> > $HADOOP_SECONDARYNAMENODE_OPTS"
> >
> > export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote
> > $HADOOP_DATANODE_OPTS"
> >
> > export HADOOP_BALANCER_OPTS="-Dcom.sun.management.jmxremote
> > $HADOOP_BALANCER_OPTS"
> >
> > export HADOOP_JOBTRACKER_OPTS="-Dcom.sun.management.jmxremote
> > $HADOOP_JOBTRACKER_OPTS"
> >
> > # export HADOOP_TASKTRACKER_OPTS=
> >
> > # The following applies to multiple commands (fs, dfs, fsck, distcp etc)
> >
> > # export HADOOP_CLIENT_OPTS
> >
> > # Extra ssh options. Empty by default.
> >
> > # export HADOOP_SSH_OPTS="-o ConnectTimeout=1 -o SendEnv=HADOOP_CONF_DIR"
> >
> > # Where log files are stored. $HADOOP_HOME/logs by default.
> >
> > # export HADOOP_LOG_DIR=${HADOOP_HOME}/logs
> >
> > # File naming remote slave hosts. $HADOOP_HOME/conf/slaves by default.
> >
> > # export HADOOP_SLAVES=${HADOOP_HOME}/conf/slaves
> >
> > # host:path where hadoop code should be rsync'd from. Unset by default.
> >
> > # export HADOOP_MASTER=master:/home/$USER/src/hadoop
> >
> > # Seconds to sleep between slave commands. Unset by default. This
> >
> > # can be useful in large clusters, where, e.g., slave rsyncs can
> >
> > # otherwise arrive faster than the master can service them.
> >
> > # export HADOOP_SLAVE_SLEEP=0.1
> >
> > # The directory where pid files are stored. /tmp by default.
> >
> > # export HADOOP_PID_DIR=/var/hadoop/pids
> >
> > # A string representing this instance of hadoop. $USER by default.
> >
> > #export HADOOP_IDENT_STRING=$USER
> >
> > # The scheduling priority for daemon processes. See 'man nice'.
> >
> > # export HADOOP_NICENESS=10
> >
> > -------------------------------------------------------------------------
> > -------------------------------------------------------------------------
Re: how to get lzo library library loaded?(error :Caused by:
java.lang.ClassNotFoundException: com.hadoop.compression.lzo.LzoCodec)
Posted by Josh Patterson <jo...@cloudera.com>.
Alex,
LZO can be a pain, we've all seen it; I have a few tips I've compiled
that might help you (I've posted these before):
http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201006.mbox/%3CAANLkTileo-q8USEiP8Y3Na9pDYHlyUFIPpR0In0LkpJm@mail.gmail.com%3E
walk through some of those and see if that does help you. I've hung
LZO up before and what has worked for me was to step through each
phase, re-doing the step as needed until I've got it working again.
Josh Patterson
Cloudera
On Sun, Aug 15, 2010 at 7:58 AM, Alex Luya <al...@gmail.com> wrote:
> Hi,
>
> At every beginning,I run:hadoop jar hadoop-*-examples.jar grep input output
> 'dfs[a-z.]+' successfully,but when run:
>
> nutch crawl url -dir crawl -depth 3,got errors:
>
> -------------------------------------------------------------------------
> -------------------------------------------------------------------------
>
> 10/08/07 22:53:30 INFO crawl.Crawl: crawl started in: crawl
>
> .....................................................................
>
> 10/08/07 22:53:30 WARN mapred.JobClient: Use GenericOptionsParser for
> parsing the arguments. Applications should implement Tool for the same.
>
> Exception in thread "main" java.lang.RuntimeException: Error in configuring
> object
>
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
>
> at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
>
> .....................................................................
>
> at org.apache.nutch.crawl.Crawl.main(Crawl.java:124)
>
> Caused by: java.lang.reflect.InvocationTargetException
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>
> .....................................................................
>
> ... 9 more
>
> Caused by: java.lang.IllegalArgumentException: Compression codec
>
> org.apache.hadoop.io.compress.GzipCodec not found.
>
> at
> org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:96)
>
> .....................................................................
>
> ... 14 more
>
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.io.compress.GzipCodec
>
> .....................................................................
>
> at
> org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:89)
>
> ... 16 more
>
> -------------------------------------------------------------------------
> -------------------------------------------------------------------------
>
> So,here GzipCode didn't get loaded successfully,or maybe it will not be
> loaded by default,I don't know,but I think it should be,then I followed this
>
> link:http://code.google.com/p/hadoop-gpl-compression/wiki/FAQ to install lzo
> and run:
>
> "nutch crawl url -dir crawl -depth 3" again,got errors:
>
> -------------------------------------------------------------------------
> -------------------------------------------------------------------------
>
> 10/08/07 22:40:41 INFO crawl.Crawl: crawl started in: crawl
>
> .....................................................................
>
> 10/08/07 22:40:42 INFO crawl.Injector: Injector: Converting injected urls to
> crawl db entries.
>
> 10/08/07 22:40:42 WARN mapred.JobClient: Use GenericOptionsParser for
> parsing the arguments. Applications should implement Tool for the same.
>
> Exception in thread "main" java.lang.RuntimeException: Error in configuring
> object
>
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
>
> .....................................................................
>
> at org.apache.nutch.crawl.Injector.inject(Injector.java:211)
>
> at org.apache.nutch.crawl.Crawl.main(Crawl.java:124)
>
> Caused by: java.lang.reflect.InvocationTargetException
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> .....................................................................
>
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
>
> ... 9 more
>
> Caused by: java.lang.IllegalArgumentException: Compression codec
>
> org.apache.hadoop.io.compress.GzipCodec not found.
>
> .....................................................................
>
> at
> org.apache.hadoop.mapred.TextInputFormat.configure(TextInputFormat.java:41)
>
> ... 14 more
>
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.io.compress.GzipCodec
>
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> .....................................................................
>
> at
> org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:89)
>
> ... 16 more
>
> -------------------------------------------------------------------------
> -------------------------------------------------------------------------
>
> run:hadoop jar hadoop-*-examples.jar grep input output 'dfs[a-z.]+',got
> errors:
>
> -------------------------------------------------------------------------
> -------------------------------------------------------------------------
>
> java.lang.RuntimeException: Error in configuring object
>
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
>
> at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
>
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
>
> at org.apache.hadoop.mapred.JobConf.getInputFormat(JobConf.java:400)
>
> .....................................................................
>
> at java.lang.reflect.Method.invoke(Method.java:597)
>
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>
> Caused by: java.lang.reflect.InvocationTargetException
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> .....................................................................
>
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
>
> ... 22 more
>
> Caused by: java.lang.IllegalArgumentException: Compression codec
> com.hadoop.compression.lzo.LzoCodec
>
> not found.
>
> at
> org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:96)
>
> at
> org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:134)
>
> at
> org.apache.hadoop.mapred.TextInputFormat.configure(TextInputFormat.java:41)
>
> ... 27 more
>
> Caused by: java.lang.ClassNotFoundException:
> com.hadoop.compression.lzo.LzoCodec
>
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> .....................................................................
>
> at
> org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:89)
>
> ... 29 more
>
> -------------------------------------------------------------------------
> -------------------------------------------------------------------------
>
> when I run " ps -aef|grep gpl",got output:
>
> -------------------------------------------------------------------------
> -------------------------------------------------------------------------
>
> alex 2267 1 1 22:04 pts/1 00:00:04 /usr/local/hadoop/jdk1.6.0_21/bin/java
> -Xmx200m -Dcom.sun.management.jmxremote
>
> -..............................................
>
> /usr/local/hadoop/hadoop-0.20.2/bin/../conf:/usr/local/hadoop/jdk1.6.0_21/lib/tools.jar:/usr/local/hadoop/hadoop-0.20.2/bin/..:/usr/local/hadoop/hadoop-0.20.2/bin/../hadoop-0.20.2-
>
> core.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/commons-cli-1.2.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/commons-
>
> codec-1.3.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/commons-el-1.0.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/commons-.-
>
> net-1.4.1.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/core-3.1.1.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/hadoop-gpl-compression-0.2.0-
>
> dev.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/jasper-
>
> compiler-5.5.12.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/jasper-
>
> runtime-5.5.12.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/jets3t-0.6.1.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/jetty-6.1.14.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/jetty-
>
> .......................................-
>
> log4j12-1.4.3.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/xmlenc-0.52.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/jsp-2.1/jsp-2.1.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/jsp-2.1/jsp-
>
> api-2.1.jar org.apache.hadoop.hdfs.server.namenode.NameNode
>
> See,two jars(hadoop-core and pgl) are existing in classpath,but seems that
> they can't be referenced by job,before this, I have tried to install
> hadoop-lzo(http://github.com/kevinweil/hadoop-lzo),same errors,maybe
> hadoop-lzo only works for hadoop 0.20,not for 0.20.1/2,I don't know.After
> one month,I haven't solved this problem,it's killing me,here I post all
> configure files,would you please help me dig problem out?Thank you.
>
> core-site.xml
>
> -------------------------------------------------------------------------
> -------------------------------------------------------------------------
>
> <configuration>
>
> <property>
>
> <name>fs.default.name</name>
>
> <value>hdfs://AlexLuya:8020</value>
>
> </property>
>
> <property>
>
> <name>hadoop.tmp.dir</name>
>
> <value>/home/alex/tmp</value>
>
> </property>
>
> <property>
>
> <name>io.compression.codecs</name>
>
> <value>org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.BZip2Codec,com.hadoop.compression.lzo.LzoCodec
>
> </value>
>
> </property>
>
> <property>
>
> <name>io.compression.codec.lzo.class</name>
>
> <value>com.hadoop.compression.lzo.LzoCodec</value>
>
> </property>
>
> </configuration>
>
> -------------------------------------------------------------------------
> -------------------------------------------------------------------------
>
> mapreduce.xml
>
> -------------------------------------------------------------------------
> -------------------------------------------------------------------------
>
> <configuration>
>
> <property>
>
> <name>mapred.job.tracker</name>
>
> <value>AlexLuya:9001</value>
>
> </property>
>
> <property>
>
> <name>mapred.tasktracker.reduce.tasks.maximum</name>
>
> <value>1</value>
>
> </property>
>
> <property>
>
> <name>mapred.tasktracker.map.tasks.maximum</name>
>
> <value>1</value>
>
> </property>
>
> <property>
>
> <name>mapred.local.dir</name>
>
> <value>/home/alex/hadoop/mapred/local</value>
>
> </property>
>
> <property>
>
> <name>mapred.system.dir</name>
>
> <value>/tmp/hadoop/mapred/system</value>
>
> </property>
>
> <property>
>
> <name>mapreduce.map.output.compress</name>
>
> <value>true</value>
>
> </property>
>
> <property>
>
> <name>mapreduce.map.output.compress.codec</name>
>
> <value>com.hadoop.compression.lzo.LzoCodec</value>
>
> </property>
>
> </configuration>
>
> -------------------------------------------------------------------------
> -------------------------------------------------------------------------
>
> hadoop-env.sh
>
> -------------------------------------------------------------------------
> -------------------------------------------------------------------------
>
> # Set Hadoop-specific environment variables here.
>
> # The only required environment variable is JAVA_HOME. All others are
>
> # optional. When running a distributed configuration it is best to
>
> # set JAVA_HOME in this file, so that it is correctly defined on
>
> # remote nodes.
>
> # The java implementation to use. Required.
>
> export JAVA_HOME=/usr/local/hadoop/jdk1.6.0_21
>
> # Extra Java CLASSPATH elements. Optional.
>
> # export HADOOP_CLASSPATH=
>
> # The maximum amount of heap to use, in MB. Default is 1000.
>
> export HADOOP_HEAPSIZE=200
>
> # Extra Java runtime options. Empty by default.
>
> #export HADOOP_OPTS=-server
>
> # Command specific options appended to HADOOP_OPTS when specified
>
> export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote
> $HADOOP_NAMENODE_OPTS"
>
> export HADOOP_SECONDARYNAMENODE_OPTS="-Dcom.sun.management.jmxremote
> $HADOOP_SECONDARYNAMENODE_OPTS"
>
> export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote
> $HADOOP_DATANODE_OPTS"
>
> export HADOOP_BALANCER_OPTS="-Dcom.sun.management.jmxremote
> $HADOOP_BALANCER_OPTS"
>
> export HADOOP_JOBTRACKER_OPTS="-Dcom.sun.management.jmxremote
> $HADOOP_JOBTRACKER_OPTS"
>
> # export HADOOP_TASKTRACKER_OPTS=
>
> # The following applies to multiple commands (fs, dfs, fsck, distcp etc)
>
> # export HADOOP_CLIENT_OPTS
>
> # Extra ssh options. Empty by default.
>
> # export HADOOP_SSH_OPTS="-o ConnectTimeout=1 -o SendEnv=HADOOP_CONF_DIR"
>
> # Where log files are stored. $HADOOP_HOME/logs by default.
>
> # export HADOOP_LOG_DIR=${HADOOP_HOME}/logs
>
> # File naming remote slave hosts. $HADOOP_HOME/conf/slaves by default.
>
> # export HADOOP_SLAVES=${HADOOP_HOME}/conf/slaves
>
> # host:path where hadoop code should be rsync'd from. Unset by default.
>
> # export HADOOP_MASTER=master:/home/$USER/src/hadoop
>
> # Seconds to sleep between slave commands. Unset by default. This
>
> # can be useful in large clusters, where, e.g., slave rsyncs can
>
> # otherwise arrive faster than the master can service them.
>
> # export HADOOP_SLAVE_SLEEP=0.1
>
> # The directory where pid files are stored. /tmp by default.
>
> # export HADOOP_PID_DIR=/var/hadoop/pids
>
> # A string representing this instance of hadoop. $USER by default.
>
> #export HADOOP_IDENT_STRING=$USER
>
> # The scheduling priority for daemon processes. See 'man nice'.
>
> # export HADOOP_NICENESS=10
>
> -------------------------------------------------------------------------
> -------------------------------------------------------------------------
Re: IOException in TaskRunner (Error Code :134)
Posted by "C.V.Krishnakumar" <cv...@me.com>.
Hi,
Thanks a lot ! I just removed the elements in the cache and the incomplete blocks, and it worked.
Regards,
Krishnakumar.
On Sep 21, 2010, at 12:54 PM, Allen Wittenauer wrote:
>
> It is a "not enough information" error.
>
> Check the tasks, jobtracker, tasktracker, datanode, and namenode logs.
>
> On Sep 21, 2010, at 12:30 PM, C.V.Krishnakumar wrote:
>
>>
>> Hi,
>> Just wanted to know if anyone has any idea about this one? This happens every time I run a job.
>> Is this issue hardware related?
>>
>> Thanks in advance,
>> Krishnakumar.
>>
>> Begin forwarded message:
>>
>>> From: "C.V.Krishnakumar" <cv...@me.com>
>>> Date: September 17, 2010 1:32:49 PM PDT
>>> To: common-user@hadoop.apache.org
>>> Subject: Tasks Failing : IOException in TaskRunner (Error Code :134)
>>> Reply-To: common-user@hadoop.apache.org
>>>
>>> Hi all,
>>>
>>> I am facing a problem with the TaskRunner. I have a small hadoop cluster in the fully distributed mode. However when I submit a job, the job never seems to proceed beyond the "map 0% reduce 0%" stage. Soon after I get this error:
>>>
>>> java.io.IOException: Task process exit with nonzero status of 134. at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418)
>>>
>>> When I googled this issue, I found this link http://markmail.org/message/lvefqbcboaqzfazt but did not completely understand the issue. The cluster, in this same configuration, was working fine a few days back. So, I am confused as to what could have changed to cause this error.
>>>
>>> Have any of you faced similar problems too? I would be really grateful if you could let me know if I am missing something very obvious.
>>>
>>> Thanks a lot!
>>>
>>> Regards,
>>> Krishna.
>>>
>>
>
Re: IOException in TaskRunner (Error Code :134)
Posted by Allen Wittenauer <aw...@linkedin.com>.
It is a "not enough information" error.
Check the tasks, jobtracker, tasktracker, datanode, and namenode logs.
On Sep 21, 2010, at 12:30 PM, C.V.Krishnakumar wrote:
>
> Hi,
> Just wanted to know if anyone has any idea about this one? This happens every time I run a job.
> Is this issue hardware related?
>
> Thanks in advance,
> Krishnakumar.
>
> Begin forwarded message:
>
>> From: "C.V.Krishnakumar" <cv...@me.com>
>> Date: September 17, 2010 1:32:49 PM PDT
>> To: common-user@hadoop.apache.org
>> Subject: Tasks Failing : IOException in TaskRunner (Error Code :134)
>> Reply-To: common-user@hadoop.apache.org
>>
>> Hi all,
>>
>> I am facing a problem with the TaskRunner. I have a small hadoop cluster in the fully distributed mode. However when I submit a job, the job never seems to proceed beyond the "map 0% reduce 0%" stage. Soon after I get this error:
>>
>> java.io.IOException: Task process exit with nonzero status of 134. at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418)
>>
>> When I googled this issue, I found this link http://markmail.org/message/lvefqbcboaqzfazt but did not completely understand the issue. The cluster, in this same configuration, was working fine a few days back. So, I am confused as to what could have changed to cause this error.
>>
>> Have any of you faced similar problems too? I would be really grateful if you could let me know if I am missing something very obvious.
>>
>> Thanks a lot!
>>
>> Regards,
>> Krishna.
>>
>
IOException in TaskRunner (Error Code :134)
Posted by "C.V.Krishnakumar" <cv...@me.com>.
Hi,
Just wanted to know if anyone has any idea about this one? This happens every time I run a job.
Is this issue hardware related?
Thanks in advance,
Krishnakumar.
Begin forwarded message:
> From: "C.V.Krishnakumar" <cv...@me.com>
> Date: September 17, 2010 1:32:49 PM PDT
> To: common-user@hadoop.apache.org
> Subject: Tasks Failing : IOException in TaskRunner (Error Code :134)
> Reply-To: common-user@hadoop.apache.org
>
> Hi all,
>
> I am facing a problem with the TaskRunner. I have a small hadoop cluster in the fully distributed mode. However when I submit a job, the job never seems to proceed beyond the "map 0% reduce 0%" stage. Soon after I get this error:
>
> java.io.IOException: Task process exit with nonzero status of 134. at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418)
>
> When I googled this issue, I found this link http://markmail.org/message/lvefqbcboaqzfazt but did not completely understand the issue. The cluster, in this same configuration, was working fine a few days back. So, I am confused as to what could have changed to cause this error.
>
> Have any of you faced similar problems too? I would be really grateful if you could let me know if I am missing something very obvious.
>
> Thanks a lot!
>
> Regards,
> Krishna.
>
Tasks Failing : IOException in TaskRunner (Error Code :134)
Posted by "C.V.Krishnakumar" <cv...@me.com>.
Hi all,
I am facing a problem with the TaskRunner. I have a small hadoop cluster in the fully distributed mode. However when I submit a job, the job never seems to proceed beyond the "map 0% reduce 0%" stage. Soon after I get this error:
java.io.IOException: Task process exit with nonzero status of 134. at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418)
When I googled this issue, I found this link http://markmail.org/message/lvefqbcboaqzfazt but did not completely understand the issue. The cluster, in this same configuration, was working fine a few days back. So, I am confused as to what could have changed to cause this error.
Have any of you faced similar problems too? I would be really grateful if you could let me know if I am missing something very obvious.
Thanks a lot!
Regards,
Krishna.