You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Margusja <ma...@roo.ee> on 2013/07/11 14:44:44 UTC
Task failure in slave node
Hi
I have tow nodes:
n1 (master, salve) and n2 (slave)
after set up I ran wordcount example and it worked fine:
[hduser@n1 ~]$ hadoop jar /usr/local/hadoop/hadoop-examples-1.0.4.jar
wordcount /user/hduser/gutenberg /user/hduser/gutenberg-output
13/07/11 15:30:44 INFO input.FileInputFormat: Total input paths to
process : 7
13/07/11 15:30:44 INFO util.NativeCodeLoader: Loaded the native-hadoop
library
13/07/11 15:30:44 WARN snappy.LoadSnappy: Snappy native library not loaded
13/07/11 15:30:44 INFO mapred.JobClient: Running job: job_201307111355_0015
13/07/11 15:30:45 INFO mapred.JobClient: map 0% reduce 0%
13/07/11 15:31:03 INFO mapred.JobClient: map 42% reduce 0%
13/07/11 15:31:06 INFO mapred.JobClient: map 57% reduce 0%
13/07/11 15:31:09 INFO mapred.JobClient: map 71% reduce 0%
13/07/11 15:31:15 INFO mapred.JobClient: map 100% reduce 0%
13/07/11 15:31:18 INFO mapred.JobClient: map 100% reduce 23%
13/07/11 15:31:27 INFO mapred.JobClient: map 100% reduce 100%
13/07/11 15:31:32 INFO mapred.JobClient: Job complete: job_201307111355_0015
13/07/11 15:31:32 INFO mapred.JobClient: Counters: 30
13/07/11 15:31:32 INFO mapred.JobClient: Job Counters
13/07/11 15:31:32 INFO mapred.JobClient: Launched reduce tasks=1
13/07/11 15:31:32 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=67576
13/07/11 15:31:32 INFO mapred.JobClient: Total time spent by all
reduces waiting after reserving slots (ms)=0
13/07/11 15:31:32 INFO mapred.JobClient: Total time spent by all
maps waiting after reserving slots (ms)=0
13/07/11 15:31:32 INFO mapred.JobClient: Rack-local map tasks=3
13/07/11 15:31:32 INFO mapred.JobClient: Launched map tasks=7
13/07/11 15:31:32 INFO mapred.JobClient: Data-local map tasks=4
13/07/11 15:31:32 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=21992
13/07/11 15:31:32 INFO mapred.JobClient: File Output Format Counters
13/07/11 15:31:32 INFO mapred.JobClient: Bytes Written=1412505
13/07/11 15:31:32 INFO mapred.JobClient: FileSystemCounters
13/07/11 15:31:32 INFO mapred.JobClient: FILE_BYTES_READ=5414195
13/07/11 15:31:32 INFO mapred.JobClient: HDFS_BYTES_READ=6950820
13/07/11 15:31:32 INFO mapred.JobClient: FILE_BYTES_WRITTEN=8744993
13/07/11 15:31:32 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=1412505
13/07/11 15:31:32 INFO mapred.JobClient: File Input Format Counters
13/07/11 15:31:32 INFO mapred.JobClient: Bytes Read=6950001
13/07/11 15:31:32 INFO mapred.JobClient: Map-Reduce Framework
13/07/11 15:31:32 INFO mapred.JobClient: Map output materialized
bytes=3157469
13/07/11 15:31:32 INFO mapred.JobClient: Map input records=137146
13/07/11 15:31:32 INFO mapred.JobClient: Reduce shuffle bytes=2904836
13/07/11 15:31:32 INFO mapred.JobClient: Spilled Records=594764
13/07/11 15:31:32 INFO mapred.JobClient: Map output bytes=11435849
13/07/11 15:31:32 INFO mapred.JobClient: Total committed heap usage
(bytes)=1128136704
13/07/11 15:31:32 INFO mapred.JobClient: CPU time spent (ms)=18230
13/07/11 15:31:32 INFO mapred.JobClient: Combine input records=1174991
13/07/11 15:31:32 INFO mapred.JobClient: SPLIT_RAW_BYTES=819
13/07/11 15:31:32 INFO mapred.JobClient: Reduce input records=218990
13/07/11 15:31:32 INFO mapred.JobClient: Reduce input groups=128513
13/07/11 15:31:32 INFO mapred.JobClient: Combine output records=218990
13/07/11 15:31:32 INFO mapred.JobClient: Physical memory (bytes)
snapshot=1179656192
13/07/11 15:31:32 INFO mapred.JobClient: Reduce output records=128513
13/07/11 15:31:32 INFO mapred.JobClient: Virtual memory (bytes)
snapshot=22992117760
13/07/11 15:31:32 INFO mapred.JobClient: Map output records=1174991
from web interface (http://n1:50030/) I saw that both (n1 and n2 ) were
used without any errors.
Problems appear if I try to use following commands in master (n1):
[hduser@n1 ~]$hadoop jar
mahout-distribution-0.7/mahout-examples-0.7-job.jar
org.apache.mahout.classifier.df.mapreduce.BuildForest
-Dmapred.max.split.size=1874231 -p -d testdata/bal_ee_2009.csv -ds
testdata/bal_ee_2009.csv.info -sl 10 -o bal_ee_2009_out -t 1
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[file:/usr/local/hadoop-1.0.4/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/local/hadoop-1.0.4/lib/slf4j-log4j12-1.4.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
13/07/11 15:36:50 INFO mapreduce.BuildForest: Partial Mapred implementation
13/07/11 15:36:50 INFO mapreduce.BuildForest: Building the forest...
13/07/11 15:36:50 WARN mapred.JobClient: No job jar file set. User
classes may not be found. See JobConf(Class) or JobConf#setJar(String).
13/07/11 15:36:50 INFO input.FileInputFormat: Total input paths to
process : 1
13/07/11 15:36:50 INFO util.NativeCodeLoader: Loaded the native-hadoop
library
13/07/11 15:36:50 WARN snappy.LoadSnappy: Snappy native library not loaded
13/07/11 15:36:50 INFO mapred.JobClient: Running job: job_201307111355_0016
13/07/11 15:36:51 INFO mapred.JobClient: map 0% reduce 0%
13/07/11 15:37:09 INFO mapred.JobClient: map 7% reduce 0%
13/07/11 15:37:11 INFO mapred.JobClient: Task Id :
attempt_201307111355_0016_m_000002_0, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException:
org.apache.mahout.classifier.df.mapreduce.partial.Step1Mapper
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:867)
at
org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:199)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:719)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.ClassNotFoundException:
org.apache.mahout.classifier.df.mapreduce.partial.Step1Mapper
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:270)
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:820)
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:865)
... 8 more
...
...
...
13/07/11 15:37:37 INFO mapred.JobClient: map 61% reduce 0%
13/07/11 15:37:46 INFO mapred.JobClient: map 76% reduce 0%
13/07/11 15:37:55 INFO mapred.JobClient: map 92% reduce 0%
13/07/11 15:38:01 INFO mapred.JobClient: map 100% reduce 0%
13/07/11 15:38:06 INFO mapred.JobClient: Job complete: job_201307111355_0016
13/07/11 15:38:06 INFO mapred.JobClient: Counters: 20
13/07/11 15:38:06 INFO mapred.JobClient: Job Counters
13/07/11 15:38:06 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=184724
13/07/11 15:38:06 INFO mapred.JobClient: Total time spent by all
reduces waiting after reserving slots (ms)=0
13/07/11 15:38:06 INFO mapred.JobClient: Total time spent by all
maps waiting after reserving slots (ms)=0
13/07/11 15:38:06 INFO mapred.JobClient: Rack-local map tasks=13
13/07/11 15:38:06 INFO mapred.JobClient: Launched map tasks=22
13/07/11 15:38:06 INFO mapred.JobClient: Data-local map tasks=9
13/07/11 15:38:06 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
13/07/11 15:38:06 INFO mapred.JobClient: File Output Format Counters
13/07/11 15:38:06 INFO mapred.JobClient: Bytes Written=184520
13/07/11 15:38:06 INFO mapred.JobClient: FileSystemCounters
13/07/11 15:38:06 INFO mapred.JobClient: HDFS_BYTES_READ=23444612
13/07/11 15:38:06 INFO mapred.JobClient: FILE_BYTES_WRITTEN=294037
13/07/11 15:38:06 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=184520
13/07/11 15:38:06 INFO mapred.JobClient: File Input Format Counters
13/07/11 15:38:06 INFO mapred.JobClient: Bytes Read=22952913
13/07/11 15:38:06 INFO mapred.JobClient: Map-Reduce Framework
13/07/11 15:38:06 INFO mapred.JobClient: Map input records=204091
13/07/11 15:38:06 INFO mapred.JobClient: Physical memory (bytes)
snapshot=1402363904
13/07/11 15:38:06 INFO mapred.JobClient: Spilled Records=0
13/07/11 15:38:06 INFO mapred.JobClient: CPU time spent (ms)=26640
13/07/11 15:38:06 INFO mapred.JobClient: Total committed heap usage
(bytes)=1333985280
13/07/11 15:38:06 INFO mapred.JobClient: Virtual memory (bytes)
snapshot=37349126144
13/07/11 15:38:06 INFO mapred.JobClient: Map output records=1
13/07/11 15:38:06 INFO mapred.JobClient: SPLIT_RAW_BYTES=1573
13/07/11 15:38:07 INFO common.HadoopUtil: Deleting
hdfs://master:54310/user/hduser/bal_ee_2009_out
13/07/11 15:38:07 INFO mapreduce.BuildForest: Build Time: 0h 1m 16s 977
13/07/11 15:38:07 INFO mapreduce.BuildForest: Forest num Nodes: 10700
13/07/11 15:38:07 INFO mapreduce.BuildForest: Forest mean num Nodes: 10700
13/07/11 15:38:07 INFO mapreduce.BuildForest: Forest mean max Depth: 17
13/07/11 15:38:07 INFO mapreduce.BuildForest: Storing the forest in:
bal_ee_2009_out/forest.seq
From web console I can see that n1 tasks are fine but n2 tasks failed:
java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.mahout.classifier.df.mapreduce.partial.Step1Mapper
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:867)
at org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:199)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:719)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.ClassNotFoundException: org.apache.mahout.classifier.df.mapreduce.partial.Step1Mapper
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:270)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:820)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:865)
... 8 more
But If I run same command from n2
[hduser@n2 ~]$ hadoop jar
mahout-distribution-0.7/mahout-examples-0.7-job.jar
org.apache.mahout.classifier.df.mapreduce.BuildForest
-Dmapred.max.split.size=1874231 -p -d testdata/bal_ee_2009.csv -ds
testdata/bal_ee_2009.csv.info -sl 10 -o bal_ee_2009_out -t 1
13/07/11 15:41:20 INFO mapreduce.BuildForest: Partial Mapred implementation
13/07/11 15:41:20 INFO mapreduce.BuildForest: Building the forest...
13/07/11 15:41:21 INFO input.FileInputFormat: Total input paths to
process : 1
13/07/11 15:41:21 INFO util.NativeCodeLoader: Loaded the native-hadoop
library
13/07/11 15:41:21 WARN snappy.LoadSnappy: Snappy native library not loaded
13/07/11 15:41:21 INFO mapred.JobClient: Running job: job_201307111355_0017
13/07/11 15:41:22 INFO mapred.JobClient: map 0% reduce 0%
13/07/11 15:41:43 INFO mapred.JobClient: map 7% reduce 0%
13/07/11 15:41:46 INFO mapred.JobClient: map 15% reduce 0%
13/07/11 15:41:49 INFO mapred.JobClient: map 38% reduce 0%
13/07/11 15:41:52 INFO mapred.JobClient: map 46% reduce 0%
13/07/11 15:42:01 INFO mapred.JobClient: map 61% reduce 0%
13/07/11 15:42:02 INFO mapred.JobClient: map 76% reduce 0%
13/07/11 15:42:09 INFO mapred.JobClient: map 84% reduce 0%
13/07/11 15:42:13 INFO mapred.JobClient: map 100% reduce 0%
13/07/11 15:42:19 INFO mapred.JobClient: Job complete: job_201307111355_0017
13/07/11 15:42:19 INFO mapred.JobClient: Counters: 20
13/07/11 15:42:19 INFO mapred.JobClient: Job Counters
13/07/11 15:42:19 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=138911
13/07/11 15:42:19 INFO mapred.JobClient: Total time spent by all
reduces waiting after reserving slots (ms)=0
13/07/11 15:42:19 INFO mapred.JobClient: Total time spent by all
maps waiting after reserving slots (ms)=0
13/07/11 15:42:19 INFO mapred.JobClient: Rack-local map tasks=6
13/07/11 15:42:19 INFO mapred.JobClient: Launched map tasks=13
13/07/11 15:42:19 INFO mapred.JobClient: Data-local map tasks=7
13/07/11 15:42:19 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
13/07/11 15:42:19 INFO mapred.JobClient: File Output Format Counters
13/07/11 15:42:19 INFO mapred.JobClient: Bytes Written=184956
13/07/11 15:42:19 INFO mapred.JobClient: FileSystemCounters
13/07/11 15:42:19 INFO mapred.JobClient: HDFS_BYTES_READ=23444612
13/07/11 15:42:19 INFO mapred.JobClient: FILE_BYTES_WRITTEN=296026
13/07/11 15:42:19 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=184956
13/07/11 15:42:19 INFO mapred.JobClient: File Input Format Counters
13/07/11 15:42:19 INFO mapred.JobClient: Bytes Read=22952913
13/07/11 15:42:19 INFO mapred.JobClient: Map-Reduce Framework
13/07/11 15:42:19 INFO mapred.JobClient: Map input records=204091
13/07/11 15:42:19 INFO mapred.JobClient: Physical memory (bytes)
snapshot=1370255360
13/07/11 15:42:19 INFO mapred.JobClient: Spilled Records=0
13/07/11 15:42:19 INFO mapred.JobClient: CPU time spent (ms)=27990
13/07/11 15:42:19 INFO mapred.JobClient: Total committed heap usage
(bytes)=1292500992
13/07/11 15:42:19 INFO mapred.JobClient: Virtual memory (bytes)
snapshot=37360082944
13/07/11 15:42:19 INFO mapred.JobClient: Map output records=1
13/07/11 15:42:19 INFO mapred.JobClient: SPLIT_RAW_BYTES=1573
13/07/11 15:42:20 INFO common.HadoopUtil: Deleting
hdfs://master:54310/user/hduser/bal_ee_2009_out
13/07/11 15:42:20 INFO mapreduce.BuildForest: Build Time: 0h 0m 59s 386
13/07/11 15:42:20 INFO mapreduce.BuildForest: Forest num Nodes: 10698
13/07/11 15:42:20 INFO mapreduce.BuildForest: Forest mean num Nodes: 10698
13/07/11 15:42:20 INFO mapreduce.BuildForest: Forest mean max Depth: 16
13/07/11 15:42:20 INFO mapreduce.BuildForest: Storing the forest in:
bal_ee_2009_out/forest.seq
Both (n1 and n2) are used and from web console I can see that there are
no errors.
Is there any explanations why I am getting errors when I run command
from master?
--
Regards, Margus (Margusja) Roo
+372 51 48 780
http://margus.roo.ee
skype: margusja
-----BEGIN PUBLIC KEY-----
MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCvbeg7LwEC2SCpAEewwpC3ajxE
5ZsRMCB77L8bae9G7TslgLkoIzo9yOjPdx2NN6DllKbV65UjTay43uUDyql9g3tl
RhiJIcoAExkSTykWqAIPR88LfilLy1JlQ+0RD8OXiWOVVQfhOHpQ0R/jcAkM2lZa
BjM8j36yJvoBVsfOHQIDAQAB
-----END PUBLIC KEY-----
Re: Task failure in slave node
Posted by Margusja <ma...@roo.ee>.
Than you, it resolved the problem.
Funny, I don't remember that I copied mahout libs to n1 hadoop but there
they are.
Tervitades, Margus (Margusja) Roo
+372 51 48 780
http://margus.roo.ee
skype: margusja
-----BEGIN PUBLIC KEY-----
MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCvbeg7LwEC2SCpAEewwpC3ajxE
5ZsRMCB77L8bae9G7TslgLkoIzo9yOjPdx2NN6DllKbV65UjTay43uUDyql9g3tl
RhiJIcoAExkSTykWqAIPR88LfilLy1JlQ+0RD8OXiWOVVQfhOHpQ0R/jcAkM2lZa
BjM8j36yJvoBVsfOHQIDAQAB
-----END PUBLIC KEY-----
On 7/11/13 4:41 PM, Azuryy Yu wrote:
>
> sorry for typo,
>
> mahout, not mahou. sent from mobile
>
> On Jul 11, 2013 9:40 PM, "Azuryy Yu" <azuryyyu@gmail.com
> <ma...@gmail.com>> wrote:
>
> hi,
>
> put all mahou jars under hadoop_home/lib, then restart cluster.
>
> On Jul 11, 2013 8:45 PM, "Margusja" <margus@roo.ee
> <ma...@roo.ee>> wrote:
>
> Hi
>
> I have tow nodes:
> n1 (master, salve) and n2 (slave)
>
> after set up I ran wordcount example and it worked fine:
> [hduser@n1 ~]$ hadoop jar
> /usr/local/hadoop/hadoop-examples-1.0.4.jar wordcount
> /user/hduser/gutenberg /user/hduser/gutenberg-output
> 13/07/11 15:30:44 INFO input.FileInputFormat: Total input
> paths to process : 7
> 13/07/11 15:30:44 INFO util.NativeCodeLoader: Loaded the
> native-hadoop library
> 13/07/11 15:30:44 WARN snappy.LoadSnappy: Snappy native
> library not loaded
> 13/07/11 15:30:44 INFO mapred.JobClient: Running job:
> job_201307111355_0015
> 13/07/11 15:30:45 INFO mapred.JobClient: map 0% reduce 0%
> 13/07/11 15:31:03 INFO mapred.JobClient: map 42% reduce 0%
> 13/07/11 15:31:06 INFO mapred.JobClient: map 57% reduce 0%
> 13/07/11 15:31:09 INFO mapred.JobClient: map 71% reduce 0%
> 13/07/11 15:31:15 INFO mapred.JobClient: map 100% reduce 0%
> 13/07/11 15:31:18 INFO mapred.JobClient: map 100% reduce 23%
> 13/07/11 15:31:27 INFO mapred.JobClient: map 100% reduce 100%
> 13/07/11 15:31:32 INFO mapred.JobClient: Job complete:
> job_201307111355_0015
> 13/07/11 15:31:32 INFO mapred.JobClient: Counters: 30
> 13/07/11 15:31:32 INFO mapred.JobClient: Job Counters
> 13/07/11 15:31:32 INFO mapred.JobClient: Launched reduce
> tasks=1
> 13/07/11 15:31:32 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=67576
> 13/07/11 15:31:32 INFO mapred.JobClient: Total time spent
> by all reduces waiting after reserving slots (ms)=0
> 13/07/11 15:31:32 INFO mapred.JobClient: Total time spent
> by all maps waiting after reserving slots (ms)=0
> 13/07/11 15:31:32 INFO mapred.JobClient: Rack-local map
> tasks=3
> 13/07/11 15:31:32 INFO mapred.JobClient: Launched map tasks=7
> 13/07/11 15:31:32 INFO mapred.JobClient: Data-local map
> tasks=4
> 13/07/11 15:31:32 INFO mapred.JobClient:
> SLOTS_MILLIS_REDUCES=21992
> 13/07/11 15:31:32 INFO mapred.JobClient: File Output Format
> Counters
> 13/07/11 15:31:32 INFO mapred.JobClient: Bytes Written=1412505
> 13/07/11 15:31:32 INFO mapred.JobClient: FileSystemCounters
> 13/07/11 15:31:32 INFO mapred.JobClient: FILE_BYTES_READ=5414195
> 13/07/11 15:31:32 INFO mapred.JobClient: HDFS_BYTES_READ=6950820
> 13/07/11 15:31:32 INFO mapred.JobClient:
> FILE_BYTES_WRITTEN=8744993
> 13/07/11 15:31:32 INFO mapred.JobClient:
> HDFS_BYTES_WRITTEN=1412505
> 13/07/11 15:31:32 INFO mapred.JobClient: File Input Format
> Counters
> 13/07/11 15:31:32 INFO mapred.JobClient: Bytes Read=6950001
> 13/07/11 15:31:32 INFO mapred.JobClient: Map-Reduce Framework
> 13/07/11 15:31:32 INFO mapred.JobClient: Map output
> materialized bytes=3157469
> 13/07/11 15:31:32 INFO mapred.JobClient: Map input
> records=137146
> 13/07/11 15:31:32 INFO mapred.JobClient: Reduce shuffle
> bytes=2904836
> 13/07/11 15:31:32 INFO mapred.JobClient: Spilled
> Records=594764
> 13/07/11 15:31:32 INFO mapred.JobClient: Map output
> bytes=11435849
> 13/07/11 15:31:32 INFO mapred.JobClient: Total committed
> heap usage (bytes)=1128136704
> 13/07/11 15:31:32 INFO mapred.JobClient: CPU time spent
> (ms)=18230
> 13/07/11 15:31:32 INFO mapred.JobClient: Combine input
> records=1174991
> 13/07/11 15:31:32 INFO mapred.JobClient: SPLIT_RAW_BYTES=819
> 13/07/11 15:31:32 INFO mapred.JobClient: Reduce input
> records=218990
> 13/07/11 15:31:32 INFO mapred.JobClient: Reduce input
> groups=128513
> 13/07/11 15:31:32 INFO mapred.JobClient: Combine output
> records=218990
> 13/07/11 15:31:32 INFO mapred.JobClient: Physical memory
> (bytes) snapshot=1179656192
> 13/07/11 15:31:32 INFO mapred.JobClient: Reduce output
> records=128513
> 13/07/11 15:31:32 INFO mapred.JobClient: Virtual memory
> (bytes) snapshot=22992117760
> 13/07/11 15:31:32 INFO mapred.JobClient: Map output
> records=1174991
>
> from web interface (http://n1:50030/) I saw that both (n1 and
> n2 ) were used without any errors.
>
> Problems appear if I try to use following commands in master (n1):
>
> [hduser@n1 ~]$hadoop jar
> mahout-distribution-0.7/mahout-examples-0.7-job.jar
> org.apache.mahout.classifier.df.mapreduce.BuildForest
> -Dmapred.max.split.size=1874231 -p -d testdata/bal_ee_2009.csv
> -ds testdata/bal_ee_2009.csv.info
> <http://bal_ee_2009.csv.info> -sl 10 -o bal_ee_2009_out -t 1
>
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [file:/usr/local/hadoop-1.0.4/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/usr/local/hadoop-1.0.4/lib/slf4j-log4j12-1.4.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings
> for an explanation.
> 13/07/11 15:36:50 INFO mapreduce.BuildForest: Partial Mapred
> implementation
> 13/07/11 15:36:50 INFO mapreduce.BuildForest: Building the
> forest...
> 13/07/11 15:36:50 WARN mapred.JobClient: No job jar file set.
> User classes may not be found. See JobConf(Class) or
> JobConf#setJar(String).
> 13/07/11 15:36:50 INFO input.FileInputFormat: Total input
> paths to process : 1
> 13/07/11 15:36:50 INFO util.NativeCodeLoader: Loaded the
> native-hadoop library
> 13/07/11 15:36:50 WARN snappy.LoadSnappy: Snappy native
> library not loaded
> 13/07/11 15:36:50 INFO mapred.JobClient: Running job:
> job_201307111355_0016
> 13/07/11 15:36:51 INFO mapred.JobClient: map 0% reduce 0%
> 13/07/11 15:37:09 INFO mapred.JobClient: map 7% reduce 0%
> 13/07/11 15:37:11 INFO mapred.JobClient: Task Id :
> attempt_201307111355_0016_m_000002_0, Status : FAILED
> java.lang.RuntimeException: java.lang.ClassNotFoundException:
> org.apache.mahout.classifier.df.mapreduce.partial.Step1Mapper
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:867)
> at
> org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:199)
> at
> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:719)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native
> Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.mahout.classifier.df.mapreduce.partial.Step1Mapper
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native
> Method)
> at
> java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:270)
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:820)
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:865)
> ... 8 more
>
> ...
> ...
> ...
>
> 13/07/11 15:37:37 INFO mapred.JobClient: map 61% reduce 0%
> 13/07/11 15:37:46 INFO mapred.JobClient: map 76% reduce 0%
> 13/07/11 15:37:55 INFO mapred.JobClient: map 92% reduce 0%
> 13/07/11 15:38:01 INFO mapred.JobClient: map 100% reduce 0%
> 13/07/11 15:38:06 INFO mapred.JobClient: Job complete:
> job_201307111355_0016
> 13/07/11 15:38:06 INFO mapred.JobClient: Counters: 20
> 13/07/11 15:38:06 INFO mapred.JobClient: Job Counters
> 13/07/11 15:38:06 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=184724
> 13/07/11 15:38:06 INFO mapred.JobClient: Total time spent
> by all reduces waiting after reserving slots (ms)=0
> 13/07/11 15:38:06 INFO mapred.JobClient: Total time spent
> by all maps waiting after reserving slots (ms)=0
> 13/07/11 15:38:06 INFO mapred.JobClient: Rack-local map
> tasks=13
> 13/07/11 15:38:06 INFO mapred.JobClient: Launched map tasks=22
> 13/07/11 15:38:06 INFO mapred.JobClient: Data-local map
> tasks=9
> 13/07/11 15:38:06 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
> 13/07/11 15:38:06 INFO mapred.JobClient: File Output Format
> Counters
> 13/07/11 15:38:06 INFO mapred.JobClient: Bytes Written=184520
> 13/07/11 15:38:06 INFO mapred.JobClient: FileSystemCounters
> 13/07/11 15:38:06 INFO mapred.JobClient: HDFS_BYTES_READ=23444612
> 13/07/11 15:38:06 INFO mapred.JobClient: FILE_BYTES_WRITTEN=294037
> 13/07/11 15:38:06 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=184520
> 13/07/11 15:38:06 INFO mapred.JobClient: File Input Format
> Counters
> 13/07/11 15:38:06 INFO mapred.JobClient: Bytes Read=22952913
> 13/07/11 15:38:06 INFO mapred.JobClient: Map-Reduce Framework
> 13/07/11 15:38:06 INFO mapred.JobClient: Map input
> records=204091
> 13/07/11 15:38:06 INFO mapred.JobClient: Physical memory
> (bytes) snapshot=1402363904
> 13/07/11 15:38:06 INFO mapred.JobClient: Spilled Records=0
> 13/07/11 15:38:06 INFO mapred.JobClient: CPU time spent
> (ms)=26640
> 13/07/11 15:38:06 INFO mapred.JobClient: Total committed
> heap usage (bytes)=1333985280
> 13/07/11 15:38:06 INFO mapred.JobClient: Virtual memory
> (bytes) snapshot=37349126144
> 13/07/11 15:38:06 INFO mapred.JobClient: Map output records=1
> 13/07/11 15:38:06 INFO mapred.JobClient: SPLIT_RAW_BYTES=1573
> 13/07/11 15:38:07 INFO common.HadoopUtil: Deleting
> hdfs://master:54310/user/hduser/bal_ee_2009_out
> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Build Time: 0h
> 1m 16s 977
> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Forest num
> Nodes: 10700
> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Forest mean num
> Nodes: 10700
> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Forest mean max
> Depth: 17
> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Storing the
> forest in: bal_ee_2009_out/forest.seq
>
> From web console I can see that n1 tasks are fine but n2 tasks
> failed:
>
> java.lang.RuntimeException: java.lang.ClassNotFoundException:
> org.apache.mahout.classifier.df.mapreduce.partial.Step1Mapper
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:867)
> at
> org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:199)
> at
> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:719)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native
> Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.mahout.classifier.df.mapreduce.partial.Step1Mapper
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native
> Method)
> at
> java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:270)
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:820)
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:865)
> ... 8 more
>
>
> But If I run same command from n2
> [hduser@n2 ~]$ hadoop jar
> mahout-distribution-0.7/mahout-examples-0.7-job.jar
> org.apache.mahout.classifier.df.mapreduce.BuildForest
> -Dmapred.max.split.size=1874231 -p -d testdata/bal_ee_2009.csv
> -ds testdata/bal_ee_2009.csv.info
> <http://bal_ee_2009.csv.info> -sl 10 -o bal_ee_2009_out -t 1
>
> 13/07/11 15:41:20 INFO mapreduce.BuildForest: Partial Mapred
> implementation
> 13/07/11 15:41:20 INFO mapreduce.BuildForest: Building the
> forest...
> 13/07/11 15:41:21 INFO input.FileInputFormat: Total input
> paths to process : 1
> 13/07/11 15:41:21 INFO util.NativeCodeLoader: Loaded the
> native-hadoop library
> 13/07/11 15:41:21 WARN snappy.LoadSnappy: Snappy native
> library not loaded
> 13/07/11 15:41:21 INFO mapred.JobClient: Running job:
> job_201307111355_0017
> 13/07/11 15:41:22 INFO mapred.JobClient: map 0% reduce 0%
> 13/07/11 15:41:43 INFO mapred.JobClient: map 7% reduce 0%
> 13/07/11 15:41:46 INFO mapred.JobClient: map 15% reduce 0%
> 13/07/11 15:41:49 INFO mapred.JobClient: map 38% reduce 0%
> 13/07/11 15:41:52 INFO mapred.JobClient: map 46% reduce 0%
> 13/07/11 15:42:01 INFO mapred.JobClient: map 61% reduce 0%
> 13/07/11 15:42:02 INFO mapred.JobClient: map 76% reduce 0%
> 13/07/11 15:42:09 INFO mapred.JobClient: map 84% reduce 0%
> 13/07/11 15:42:13 INFO mapred.JobClient: map 100% reduce 0%
> 13/07/11 15:42:19 INFO mapred.JobClient: Job complete:
> job_201307111355_0017
> 13/07/11 15:42:19 INFO mapred.JobClient: Counters: 20
> 13/07/11 15:42:19 INFO mapred.JobClient: Job Counters
> 13/07/11 15:42:19 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=138911
> 13/07/11 15:42:19 INFO mapred.JobClient: Total time spent
> by all reduces waiting after reserving slots (ms)=0
> 13/07/11 15:42:19 INFO mapred.JobClient: Total time spent
> by all maps waiting after reserving slots (ms)=0
> 13/07/11 15:42:19 INFO mapred.JobClient: Rack-local map
> tasks=6
> 13/07/11 15:42:19 INFO mapred.JobClient: Launched map tasks=13
> 13/07/11 15:42:19 INFO mapred.JobClient: Data-local map
> tasks=7
> 13/07/11 15:42:19 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
> 13/07/11 15:42:19 INFO mapred.JobClient: File Output Format
> Counters
> 13/07/11 15:42:19 INFO mapred.JobClient: Bytes Written=184956
> 13/07/11 15:42:19 INFO mapred.JobClient: FileSystemCounters
> 13/07/11 15:42:19 INFO mapred.JobClient: HDFS_BYTES_READ=23444612
> 13/07/11 15:42:19 INFO mapred.JobClient: FILE_BYTES_WRITTEN=296026
> 13/07/11 15:42:19 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=184956
> 13/07/11 15:42:19 INFO mapred.JobClient: File Input Format
> Counters
> 13/07/11 15:42:19 INFO mapred.JobClient: Bytes Read=22952913
> 13/07/11 15:42:19 INFO mapred.JobClient: Map-Reduce Framework
> 13/07/11 15:42:19 INFO mapred.JobClient: Map input
> records=204091
> 13/07/11 15:42:19 INFO mapred.JobClient: Physical memory
> (bytes) snapshot=1370255360
> 13/07/11 15:42:19 INFO mapred.JobClient: Spilled Records=0
> 13/07/11 15:42:19 INFO mapred.JobClient: CPU time spent
> (ms)=27990
> 13/07/11 15:42:19 INFO mapred.JobClient: Total committed
> heap usage (bytes)=1292500992
> 13/07/11 15:42:19 INFO mapred.JobClient: Virtual memory
> (bytes) snapshot=37360082944
> 13/07/11 15:42:19 INFO mapred.JobClient: Map output records=1
> 13/07/11 15:42:19 INFO mapred.JobClient: SPLIT_RAW_BYTES=1573
> 13/07/11 15:42:20 INFO common.HadoopUtil: Deleting
> hdfs://master:54310/user/hduser/bal_ee_2009_out
> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Build Time: 0h
> 0m 59s 386
> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Forest num
> Nodes: 10698
> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Forest mean num
> Nodes: 10698
> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Forest mean max
> Depth: 16
> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Storing the
> forest in: bal_ee_2009_out/forest.seq
>
> Both (n1 and n2) are used and from web console I can see that
> there are no errors.
>
> Is there any explanations why I am getting errors when I run
> command from master?
>
>
> --
> Regards, Margus (Margusja) Roo
> +372 51 48 780
> http://margus.roo.ee
> skype: margusja
> -----BEGIN PUBLIC KEY-----
> MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCvbeg7LwEC2SCpAEewwpC3ajxE
> 5ZsRMCB77L8bae9G7TslgLkoIzo9yOjPdx2NN6DllKbV65UjTay43uUDyql9g3tl
> RhiJIcoAExkSTykWqAIPR88LfilLy1JlQ+0RD8OXiWOVVQfhOHpQ0R/jcAkM2lZa
> BjM8j36yJvoBVsfOHQIDAQAB
> -----END PUBLIC KEY-----
>
Re: Task failure in slave node
Posted by Margusja <ma...@roo.ee>.
Than you, it resolved the problem.
Funny, I don't remember that I copied mahout libs to n1 hadoop but there
they are.
Tervitades, Margus (Margusja) Roo
+372 51 48 780
http://margus.roo.ee
skype: margusja
-----BEGIN PUBLIC KEY-----
MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCvbeg7LwEC2SCpAEewwpC3ajxE
5ZsRMCB77L8bae9G7TslgLkoIzo9yOjPdx2NN6DllKbV65UjTay43uUDyql9g3tl
RhiJIcoAExkSTykWqAIPR88LfilLy1JlQ+0RD8OXiWOVVQfhOHpQ0R/jcAkM2lZa
BjM8j36yJvoBVsfOHQIDAQAB
-----END PUBLIC KEY-----
On 7/11/13 4:41 PM, Azuryy Yu wrote:
>
> sorry for typo,
>
> mahout, not mahou. sent from mobile
>
> On Jul 11, 2013 9:40 PM, "Azuryy Yu" <azuryyyu@gmail.com
> <ma...@gmail.com>> wrote:
>
> hi,
>
> put all mahou jars under hadoop_home/lib, then restart cluster.
>
> On Jul 11, 2013 8:45 PM, "Margusja" <margus@roo.ee
> <ma...@roo.ee>> wrote:
>
> Hi
>
> I have tow nodes:
> n1 (master, salve) and n2 (slave)
>
> after set up I ran wordcount example and it worked fine:
> [hduser@n1 ~]$ hadoop jar
> /usr/local/hadoop/hadoop-examples-1.0.4.jar wordcount
> /user/hduser/gutenberg /user/hduser/gutenberg-output
> 13/07/11 15:30:44 INFO input.FileInputFormat: Total input
> paths to process : 7
> 13/07/11 15:30:44 INFO util.NativeCodeLoader: Loaded the
> native-hadoop library
> 13/07/11 15:30:44 WARN snappy.LoadSnappy: Snappy native
> library not loaded
> 13/07/11 15:30:44 INFO mapred.JobClient: Running job:
> job_201307111355_0015
> 13/07/11 15:30:45 INFO mapred.JobClient: map 0% reduce 0%
> 13/07/11 15:31:03 INFO mapred.JobClient: map 42% reduce 0%
> 13/07/11 15:31:06 INFO mapred.JobClient: map 57% reduce 0%
> 13/07/11 15:31:09 INFO mapred.JobClient: map 71% reduce 0%
> 13/07/11 15:31:15 INFO mapred.JobClient: map 100% reduce 0%
> 13/07/11 15:31:18 INFO mapred.JobClient: map 100% reduce 23%
> 13/07/11 15:31:27 INFO mapred.JobClient: map 100% reduce 100%
> 13/07/11 15:31:32 INFO mapred.JobClient: Job complete:
> job_201307111355_0015
> 13/07/11 15:31:32 INFO mapred.JobClient: Counters: 30
> 13/07/11 15:31:32 INFO mapred.JobClient: Job Counters
> 13/07/11 15:31:32 INFO mapred.JobClient: Launched reduce
> tasks=1
> 13/07/11 15:31:32 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=67576
> 13/07/11 15:31:32 INFO mapred.JobClient: Total time spent
> by all reduces waiting after reserving slots (ms)=0
> 13/07/11 15:31:32 INFO mapred.JobClient: Total time spent
> by all maps waiting after reserving slots (ms)=0
> 13/07/11 15:31:32 INFO mapred.JobClient: Rack-local map
> tasks=3
> 13/07/11 15:31:32 INFO mapred.JobClient: Launched map tasks=7
> 13/07/11 15:31:32 INFO mapred.JobClient: Data-local map
> tasks=4
> 13/07/11 15:31:32 INFO mapred.JobClient:
> SLOTS_MILLIS_REDUCES=21992
> 13/07/11 15:31:32 INFO mapred.JobClient: File Output Format
> Counters
> 13/07/11 15:31:32 INFO mapred.JobClient: Bytes Written=1412505
> 13/07/11 15:31:32 INFO mapred.JobClient: FileSystemCounters
> 13/07/11 15:31:32 INFO mapred.JobClient: FILE_BYTES_READ=5414195
> 13/07/11 15:31:32 INFO mapred.JobClient: HDFS_BYTES_READ=6950820
> 13/07/11 15:31:32 INFO mapred.JobClient:
> FILE_BYTES_WRITTEN=8744993
> 13/07/11 15:31:32 INFO mapred.JobClient:
> HDFS_BYTES_WRITTEN=1412505
> 13/07/11 15:31:32 INFO mapred.JobClient: File Input Format
> Counters
> 13/07/11 15:31:32 INFO mapred.JobClient: Bytes Read=6950001
> 13/07/11 15:31:32 INFO mapred.JobClient: Map-Reduce Framework
> 13/07/11 15:31:32 INFO mapred.JobClient: Map output
> materialized bytes=3157469
> 13/07/11 15:31:32 INFO mapred.JobClient: Map input
> records=137146
> 13/07/11 15:31:32 INFO mapred.JobClient: Reduce shuffle
> bytes=2904836
> 13/07/11 15:31:32 INFO mapred.JobClient: Spilled
> Records=594764
> 13/07/11 15:31:32 INFO mapred.JobClient: Map output
> bytes=11435849
> 13/07/11 15:31:32 INFO mapred.JobClient: Total committed
> heap usage (bytes)=1128136704
> 13/07/11 15:31:32 INFO mapred.JobClient: CPU time spent
> (ms)=18230
> 13/07/11 15:31:32 INFO mapred.JobClient: Combine input
> records=1174991
> 13/07/11 15:31:32 INFO mapred.JobClient: SPLIT_RAW_BYTES=819
> 13/07/11 15:31:32 INFO mapred.JobClient: Reduce input
> records=218990
> 13/07/11 15:31:32 INFO mapred.JobClient: Reduce input
> groups=128513
> 13/07/11 15:31:32 INFO mapred.JobClient: Combine output
> records=218990
> 13/07/11 15:31:32 INFO mapred.JobClient: Physical memory
> (bytes) snapshot=1179656192
> 13/07/11 15:31:32 INFO mapred.JobClient: Reduce output
> records=128513
> 13/07/11 15:31:32 INFO mapred.JobClient: Virtual memory
> (bytes) snapshot=22992117760
> 13/07/11 15:31:32 INFO mapred.JobClient: Map output
> records=1174991
>
> from web interface (http://n1:50030/) I saw that both (n1 and
> n2 ) were used without any errors.
>
> Problems appear if I try to use following commands in master (n1):
>
> [hduser@n1 ~]$hadoop jar
> mahout-distribution-0.7/mahout-examples-0.7-job.jar
> org.apache.mahout.classifier.df.mapreduce.BuildForest
> -Dmapred.max.split.size=1874231 -p -d testdata/bal_ee_2009.csv
> -ds testdata/bal_ee_2009.csv.info
> <http://bal_ee_2009.csv.info> -sl 10 -o bal_ee_2009_out -t 1
>
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [file:/usr/local/hadoop-1.0.4/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/usr/local/hadoop-1.0.4/lib/slf4j-log4j12-1.4.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings
> for an explanation.
> 13/07/11 15:36:50 INFO mapreduce.BuildForest: Partial Mapred
> implementation
> 13/07/11 15:36:50 INFO mapreduce.BuildForest: Building the
> forest...
> 13/07/11 15:36:50 WARN mapred.JobClient: No job jar file set.
> User classes may not be found. See JobConf(Class) or
> JobConf#setJar(String).
> 13/07/11 15:36:50 INFO input.FileInputFormat: Total input
> paths to process : 1
> 13/07/11 15:36:50 INFO util.NativeCodeLoader: Loaded the
> native-hadoop library
> 13/07/11 15:36:50 WARN snappy.LoadSnappy: Snappy native
> library not loaded
> 13/07/11 15:36:50 INFO mapred.JobClient: Running job:
> job_201307111355_0016
> 13/07/11 15:36:51 INFO mapred.JobClient: map 0% reduce 0%
> 13/07/11 15:37:09 INFO mapred.JobClient: map 7% reduce 0%
> 13/07/11 15:37:11 INFO mapred.JobClient: Task Id :
> attempt_201307111355_0016_m_000002_0, Status : FAILED
> java.lang.RuntimeException: java.lang.ClassNotFoundException:
> org.apache.mahout.classifier.df.mapreduce.partial.Step1Mapper
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:867)
> at
> org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:199)
> at
> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:719)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native
> Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.mahout.classifier.df.mapreduce.partial.Step1Mapper
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native
> Method)
> at
> java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:270)
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:820)
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:865)
> ... 8 more
>
> ...
> ...
> ...
>
> 13/07/11 15:37:37 INFO mapred.JobClient: map 61% reduce 0%
> 13/07/11 15:37:46 INFO mapred.JobClient: map 76% reduce 0%
> 13/07/11 15:37:55 INFO mapred.JobClient: map 92% reduce 0%
> 13/07/11 15:38:01 INFO mapred.JobClient: map 100% reduce 0%
> 13/07/11 15:38:06 INFO mapred.JobClient: Job complete:
> job_201307111355_0016
> 13/07/11 15:38:06 INFO mapred.JobClient: Counters: 20
> 13/07/11 15:38:06 INFO mapred.JobClient: Job Counters
> 13/07/11 15:38:06 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=184724
> 13/07/11 15:38:06 INFO mapred.JobClient: Total time spent
> by all reduces waiting after reserving slots (ms)=0
> 13/07/11 15:38:06 INFO mapred.JobClient: Total time spent
> by all maps waiting after reserving slots (ms)=0
> 13/07/11 15:38:06 INFO mapred.JobClient: Rack-local map
> tasks=13
> 13/07/11 15:38:06 INFO mapred.JobClient: Launched map tasks=22
> 13/07/11 15:38:06 INFO mapred.JobClient: Data-local map
> tasks=9
> 13/07/11 15:38:06 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
> 13/07/11 15:38:06 INFO mapred.JobClient: File Output Format
> Counters
> 13/07/11 15:38:06 INFO mapred.JobClient: Bytes Written=184520
> 13/07/11 15:38:06 INFO mapred.JobClient: FileSystemCounters
> 13/07/11 15:38:06 INFO mapred.JobClient: HDFS_BYTES_READ=23444612
> 13/07/11 15:38:06 INFO mapred.JobClient: FILE_BYTES_WRITTEN=294037
> 13/07/11 15:38:06 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=184520
> 13/07/11 15:38:06 INFO mapred.JobClient: File Input Format
> Counters
> 13/07/11 15:38:06 INFO mapred.JobClient: Bytes Read=22952913
> 13/07/11 15:38:06 INFO mapred.JobClient: Map-Reduce Framework
> 13/07/11 15:38:06 INFO mapred.JobClient: Map input
> records=204091
> 13/07/11 15:38:06 INFO mapred.JobClient: Physical memory
> (bytes) snapshot=1402363904
> 13/07/11 15:38:06 INFO mapred.JobClient: Spilled Records=0
> 13/07/11 15:38:06 INFO mapred.JobClient: CPU time spent
> (ms)=26640
> 13/07/11 15:38:06 INFO mapred.JobClient: Total committed
> heap usage (bytes)=1333985280
> 13/07/11 15:38:06 INFO mapred.JobClient: Virtual memory
> (bytes) snapshot=37349126144
> 13/07/11 15:38:06 INFO mapred.JobClient: Map output records=1
> 13/07/11 15:38:06 INFO mapred.JobClient: SPLIT_RAW_BYTES=1573
> 13/07/11 15:38:07 INFO common.HadoopUtil: Deleting
> hdfs://master:54310/user/hduser/bal_ee_2009_out
> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Build Time: 0h
> 1m 16s 977
> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Forest num
> Nodes: 10700
> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Forest mean num
> Nodes: 10700
> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Forest mean max
> Depth: 17
> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Storing the
> forest in: bal_ee_2009_out/forest.seq
>
> From web console I can see that n1 tasks are fine but n2 tasks
> failed:
>
> java.lang.RuntimeException: java.lang.ClassNotFoundException:
> org.apache.mahout.classifier.df.mapreduce.partial.Step1Mapper
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:867)
> at
> org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:199)
> at
> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:719)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native
> Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.mahout.classifier.df.mapreduce.partial.Step1Mapper
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native
> Method)
> at
> java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:270)
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:820)
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:865)
> ... 8 more
>
>
> But If I run same command from n2
> [hduser@n2 ~]$ hadoop jar
> mahout-distribution-0.7/mahout-examples-0.7-job.jar
> org.apache.mahout.classifier.df.mapreduce.BuildForest
> -Dmapred.max.split.size=1874231 -p -d testdata/bal_ee_2009.csv
> -ds testdata/bal_ee_2009.csv.info
> <http://bal_ee_2009.csv.info> -sl 10 -o bal_ee_2009_out -t 1
>
> 13/07/11 15:41:20 INFO mapreduce.BuildForest: Partial Mapred
> implementation
> 13/07/11 15:41:20 INFO mapreduce.BuildForest: Building the
> forest...
> 13/07/11 15:41:21 INFO input.FileInputFormat: Total input
> paths to process : 1
> 13/07/11 15:41:21 INFO util.NativeCodeLoader: Loaded the
> native-hadoop library
> 13/07/11 15:41:21 WARN snappy.LoadSnappy: Snappy native
> library not loaded
> 13/07/11 15:41:21 INFO mapred.JobClient: Running job:
> job_201307111355_0017
> 13/07/11 15:41:22 INFO mapred.JobClient: map 0% reduce 0%
> 13/07/11 15:41:43 INFO mapred.JobClient: map 7% reduce 0%
> 13/07/11 15:41:46 INFO mapred.JobClient: map 15% reduce 0%
> 13/07/11 15:41:49 INFO mapred.JobClient: map 38% reduce 0%
> 13/07/11 15:41:52 INFO mapred.JobClient: map 46% reduce 0%
> 13/07/11 15:42:01 INFO mapred.JobClient: map 61% reduce 0%
> 13/07/11 15:42:02 INFO mapred.JobClient: map 76% reduce 0%
> 13/07/11 15:42:09 INFO mapred.JobClient: map 84% reduce 0%
> 13/07/11 15:42:13 INFO mapred.JobClient: map 100% reduce 0%
> 13/07/11 15:42:19 INFO mapred.JobClient: Job complete:
> job_201307111355_0017
> 13/07/11 15:42:19 INFO mapred.JobClient: Counters: 20
> 13/07/11 15:42:19 INFO mapred.JobClient: Job Counters
> 13/07/11 15:42:19 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=138911
> 13/07/11 15:42:19 INFO mapred.JobClient: Total time spent
> by all reduces waiting after reserving slots (ms)=0
> 13/07/11 15:42:19 INFO mapred.JobClient: Total time spent
> by all maps waiting after reserving slots (ms)=0
> 13/07/11 15:42:19 INFO mapred.JobClient: Rack-local map
> tasks=6
> 13/07/11 15:42:19 INFO mapred.JobClient: Launched map tasks=13
> 13/07/11 15:42:19 INFO mapred.JobClient: Data-local map
> tasks=7
> 13/07/11 15:42:19 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
> 13/07/11 15:42:19 INFO mapred.JobClient: File Output Format
> Counters
> 13/07/11 15:42:19 INFO mapred.JobClient: Bytes Written=184956
> 13/07/11 15:42:19 INFO mapred.JobClient: FileSystemCounters
> 13/07/11 15:42:19 INFO mapred.JobClient: HDFS_BYTES_READ=23444612
> 13/07/11 15:42:19 INFO mapred.JobClient: FILE_BYTES_WRITTEN=296026
> 13/07/11 15:42:19 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=184956
> 13/07/11 15:42:19 INFO mapred.JobClient: File Input Format
> Counters
> 13/07/11 15:42:19 INFO mapred.JobClient: Bytes Read=22952913
> 13/07/11 15:42:19 INFO mapred.JobClient: Map-Reduce Framework
> 13/07/11 15:42:19 INFO mapred.JobClient: Map input
> records=204091
> 13/07/11 15:42:19 INFO mapred.JobClient: Physical memory
> (bytes) snapshot=1370255360
> 13/07/11 15:42:19 INFO mapred.JobClient: Spilled Records=0
> 13/07/11 15:42:19 INFO mapred.JobClient: CPU time spent
> (ms)=27990
> 13/07/11 15:42:19 INFO mapred.JobClient: Total committed
> heap usage (bytes)=1292500992
> 13/07/11 15:42:19 INFO mapred.JobClient: Virtual memory
> (bytes) snapshot=37360082944
> 13/07/11 15:42:19 INFO mapred.JobClient: Map output records=1
> 13/07/11 15:42:19 INFO mapred.JobClient: SPLIT_RAW_BYTES=1573
> 13/07/11 15:42:20 INFO common.HadoopUtil: Deleting
> hdfs://master:54310/user/hduser/bal_ee_2009_out
> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Build Time: 0h
> 0m 59s 386
> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Forest num
> Nodes: 10698
> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Forest mean num
> Nodes: 10698
> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Forest mean max
> Depth: 16
> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Storing the
> forest in: bal_ee_2009_out/forest.seq
>
> Both (n1 and n2) are used and from web console I can see that
> there are no errors.
>
> Is there any explanations why I am getting errors when I run
> command from master?
>
>
> --
> Regards, Margus (Margusja) Roo
> +372 51 48 780
> http://margus.roo.ee
> skype: margusja
> -----BEGIN PUBLIC KEY-----
> MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCvbeg7LwEC2SCpAEewwpC3ajxE
> 5ZsRMCB77L8bae9G7TslgLkoIzo9yOjPdx2NN6DllKbV65UjTay43uUDyql9g3tl
> RhiJIcoAExkSTykWqAIPR88LfilLy1JlQ+0RD8OXiWOVVQfhOHpQ0R/jcAkM2lZa
> BjM8j36yJvoBVsfOHQIDAQAB
> -----END PUBLIC KEY-----
>
Re: Task failure in slave node
Posted by Margusja <ma...@roo.ee>.
Than you, it resolved the problem.
Funny, I don't remember that I copied mahout libs to n1 hadoop but there
they are.
Tervitades, Margus (Margusja) Roo
+372 51 48 780
http://margus.roo.ee
skype: margusja
-----BEGIN PUBLIC KEY-----
MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCvbeg7LwEC2SCpAEewwpC3ajxE
5ZsRMCB77L8bae9G7TslgLkoIzo9yOjPdx2NN6DllKbV65UjTay43uUDyql9g3tl
RhiJIcoAExkSTykWqAIPR88LfilLy1JlQ+0RD8OXiWOVVQfhOHpQ0R/jcAkM2lZa
BjM8j36yJvoBVsfOHQIDAQAB
-----END PUBLIC KEY-----
On 7/11/13 4:41 PM, Azuryy Yu wrote:
>
> sorry for typo,
>
> mahout, not mahou. sent from mobile
>
> On Jul 11, 2013 9:40 PM, "Azuryy Yu" <azuryyyu@gmail.com
> <ma...@gmail.com>> wrote:
>
> hi,
>
> put all mahou jars under hadoop_home/lib, then restart cluster.
>
> On Jul 11, 2013 8:45 PM, "Margusja" <margus@roo.ee
> <ma...@roo.ee>> wrote:
>
> Hi
>
> I have tow nodes:
> n1 (master, salve) and n2 (slave)
>
> after set up I ran wordcount example and it worked fine:
> [hduser@n1 ~]$ hadoop jar
> /usr/local/hadoop/hadoop-examples-1.0.4.jar wordcount
> /user/hduser/gutenberg /user/hduser/gutenberg-output
> 13/07/11 15:30:44 INFO input.FileInputFormat: Total input
> paths to process : 7
> 13/07/11 15:30:44 INFO util.NativeCodeLoader: Loaded the
> native-hadoop library
> 13/07/11 15:30:44 WARN snappy.LoadSnappy: Snappy native
> library not loaded
> 13/07/11 15:30:44 INFO mapred.JobClient: Running job:
> job_201307111355_0015
> 13/07/11 15:30:45 INFO mapred.JobClient: map 0% reduce 0%
> 13/07/11 15:31:03 INFO mapred.JobClient: map 42% reduce 0%
> 13/07/11 15:31:06 INFO mapred.JobClient: map 57% reduce 0%
> 13/07/11 15:31:09 INFO mapred.JobClient: map 71% reduce 0%
> 13/07/11 15:31:15 INFO mapred.JobClient: map 100% reduce 0%
> 13/07/11 15:31:18 INFO mapred.JobClient: map 100% reduce 23%
> 13/07/11 15:31:27 INFO mapred.JobClient: map 100% reduce 100%
> 13/07/11 15:31:32 INFO mapred.JobClient: Job complete:
> job_201307111355_0015
> 13/07/11 15:31:32 INFO mapred.JobClient: Counters: 30
> 13/07/11 15:31:32 INFO mapred.JobClient: Job Counters
> 13/07/11 15:31:32 INFO mapred.JobClient: Launched reduce
> tasks=1
> 13/07/11 15:31:32 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=67576
> 13/07/11 15:31:32 INFO mapred.JobClient: Total time spent
> by all reduces waiting after reserving slots (ms)=0
> 13/07/11 15:31:32 INFO mapred.JobClient: Total time spent
> by all maps waiting after reserving slots (ms)=0
> 13/07/11 15:31:32 INFO mapred.JobClient: Rack-local map
> tasks=3
> 13/07/11 15:31:32 INFO mapred.JobClient: Launched map tasks=7
> 13/07/11 15:31:32 INFO mapred.JobClient: Data-local map
> tasks=4
> 13/07/11 15:31:32 INFO mapred.JobClient:
> SLOTS_MILLIS_REDUCES=21992
> 13/07/11 15:31:32 INFO mapred.JobClient: File Output Format
> Counters
> 13/07/11 15:31:32 INFO mapred.JobClient: Bytes Written=1412505
> 13/07/11 15:31:32 INFO mapred.JobClient: FileSystemCounters
> 13/07/11 15:31:32 INFO mapred.JobClient: FILE_BYTES_READ=5414195
> 13/07/11 15:31:32 INFO mapred.JobClient: HDFS_BYTES_READ=6950820
> 13/07/11 15:31:32 INFO mapred.JobClient:
> FILE_BYTES_WRITTEN=8744993
> 13/07/11 15:31:32 INFO mapred.JobClient:
> HDFS_BYTES_WRITTEN=1412505
> 13/07/11 15:31:32 INFO mapred.JobClient: File Input Format
> Counters
> 13/07/11 15:31:32 INFO mapred.JobClient: Bytes Read=6950001
> 13/07/11 15:31:32 INFO mapred.JobClient: Map-Reduce Framework
> 13/07/11 15:31:32 INFO mapred.JobClient: Map output
> materialized bytes=3157469
> 13/07/11 15:31:32 INFO mapred.JobClient: Map input
> records=137146
> 13/07/11 15:31:32 INFO mapred.JobClient: Reduce shuffle
> bytes=2904836
> 13/07/11 15:31:32 INFO mapred.JobClient: Spilled
> Records=594764
> 13/07/11 15:31:32 INFO mapred.JobClient: Map output
> bytes=11435849
> 13/07/11 15:31:32 INFO mapred.JobClient: Total committed
> heap usage (bytes)=1128136704
> 13/07/11 15:31:32 INFO mapred.JobClient: CPU time spent
> (ms)=18230
> 13/07/11 15:31:32 INFO mapred.JobClient: Combine input
> records=1174991
> 13/07/11 15:31:32 INFO mapred.JobClient: SPLIT_RAW_BYTES=819
> 13/07/11 15:31:32 INFO mapred.JobClient: Reduce input
> records=218990
> 13/07/11 15:31:32 INFO mapred.JobClient: Reduce input
> groups=128513
> 13/07/11 15:31:32 INFO mapred.JobClient: Combine output
> records=218990
> 13/07/11 15:31:32 INFO mapred.JobClient: Physical memory
> (bytes) snapshot=1179656192
> 13/07/11 15:31:32 INFO mapred.JobClient: Reduce output
> records=128513
> 13/07/11 15:31:32 INFO mapred.JobClient: Virtual memory
> (bytes) snapshot=22992117760
> 13/07/11 15:31:32 INFO mapred.JobClient: Map output
> records=1174991
>
> from web interface (http://n1:50030/) I saw that both (n1 and
> n2 ) were used without any errors.
>
> Problems appear if I try to use following commands in master (n1):
>
> [hduser@n1 ~]$hadoop jar
> mahout-distribution-0.7/mahout-examples-0.7-job.jar
> org.apache.mahout.classifier.df.mapreduce.BuildForest
> -Dmapred.max.split.size=1874231 -p -d testdata/bal_ee_2009.csv
> -ds testdata/bal_ee_2009.csv.info
> <http://bal_ee_2009.csv.info> -sl 10 -o bal_ee_2009_out -t 1
>
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [file:/usr/local/hadoop-1.0.4/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/usr/local/hadoop-1.0.4/lib/slf4j-log4j12-1.4.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings
> for an explanation.
> 13/07/11 15:36:50 INFO mapreduce.BuildForest: Partial Mapred
> implementation
> 13/07/11 15:36:50 INFO mapreduce.BuildForest: Building the
> forest...
> 13/07/11 15:36:50 WARN mapred.JobClient: No job jar file set.
> User classes may not be found. See JobConf(Class) or
> JobConf#setJar(String).
> 13/07/11 15:36:50 INFO input.FileInputFormat: Total input
> paths to process : 1
> 13/07/11 15:36:50 INFO util.NativeCodeLoader: Loaded the
> native-hadoop library
> 13/07/11 15:36:50 WARN snappy.LoadSnappy: Snappy native
> library not loaded
> 13/07/11 15:36:50 INFO mapred.JobClient: Running job:
> job_201307111355_0016
> 13/07/11 15:36:51 INFO mapred.JobClient: map 0% reduce 0%
> 13/07/11 15:37:09 INFO mapred.JobClient: map 7% reduce 0%
> 13/07/11 15:37:11 INFO mapred.JobClient: Task Id :
> attempt_201307111355_0016_m_000002_0, Status : FAILED
> java.lang.RuntimeException: java.lang.ClassNotFoundException:
> org.apache.mahout.classifier.df.mapreduce.partial.Step1Mapper
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:867)
> at
> org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:199)
> at
> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:719)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native
> Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.mahout.classifier.df.mapreduce.partial.Step1Mapper
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native
> Method)
> at
> java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:270)
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:820)
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:865)
> ... 8 more
>
> ...
> ...
> ...
>
> 13/07/11 15:37:37 INFO mapred.JobClient: map 61% reduce 0%
> 13/07/11 15:37:46 INFO mapred.JobClient: map 76% reduce 0%
> 13/07/11 15:37:55 INFO mapred.JobClient: map 92% reduce 0%
> 13/07/11 15:38:01 INFO mapred.JobClient: map 100% reduce 0%
> 13/07/11 15:38:06 INFO mapred.JobClient: Job complete:
> job_201307111355_0016
> 13/07/11 15:38:06 INFO mapred.JobClient: Counters: 20
> 13/07/11 15:38:06 INFO mapred.JobClient: Job Counters
> 13/07/11 15:38:06 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=184724
> 13/07/11 15:38:06 INFO mapred.JobClient: Total time spent
> by all reduces waiting after reserving slots (ms)=0
> 13/07/11 15:38:06 INFO mapred.JobClient: Total time spent
> by all maps waiting after reserving slots (ms)=0
> 13/07/11 15:38:06 INFO mapred.JobClient: Rack-local map
> tasks=13
> 13/07/11 15:38:06 INFO mapred.JobClient: Launched map tasks=22
> 13/07/11 15:38:06 INFO mapred.JobClient: Data-local map
> tasks=9
> 13/07/11 15:38:06 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
> 13/07/11 15:38:06 INFO mapred.JobClient: File Output Format
> Counters
> 13/07/11 15:38:06 INFO mapred.JobClient: Bytes Written=184520
> 13/07/11 15:38:06 INFO mapred.JobClient: FileSystemCounters
> 13/07/11 15:38:06 INFO mapred.JobClient: HDFS_BYTES_READ=23444612
> 13/07/11 15:38:06 INFO mapred.JobClient: FILE_BYTES_WRITTEN=294037
> 13/07/11 15:38:06 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=184520
> 13/07/11 15:38:06 INFO mapred.JobClient: File Input Format
> Counters
> 13/07/11 15:38:06 INFO mapred.JobClient: Bytes Read=22952913
> 13/07/11 15:38:06 INFO mapred.JobClient: Map-Reduce Framework
> 13/07/11 15:38:06 INFO mapred.JobClient: Map input
> records=204091
> 13/07/11 15:38:06 INFO mapred.JobClient: Physical memory
> (bytes) snapshot=1402363904
> 13/07/11 15:38:06 INFO mapred.JobClient: Spilled Records=0
> 13/07/11 15:38:06 INFO mapred.JobClient: CPU time spent
> (ms)=26640
> 13/07/11 15:38:06 INFO mapred.JobClient: Total committed
> heap usage (bytes)=1333985280
> 13/07/11 15:38:06 INFO mapred.JobClient: Virtual memory
> (bytes) snapshot=37349126144
> 13/07/11 15:38:06 INFO mapred.JobClient: Map output records=1
> 13/07/11 15:38:06 INFO mapred.JobClient: SPLIT_RAW_BYTES=1573
> 13/07/11 15:38:07 INFO common.HadoopUtil: Deleting
> hdfs://master:54310/user/hduser/bal_ee_2009_out
> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Build Time: 0h
> 1m 16s 977
> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Forest num
> Nodes: 10700
> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Forest mean num
> Nodes: 10700
> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Forest mean max
> Depth: 17
> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Storing the
> forest in: bal_ee_2009_out/forest.seq
>
> From web console I can see that n1 tasks are fine but n2 tasks
> failed:
>
> java.lang.RuntimeException: java.lang.ClassNotFoundException:
> org.apache.mahout.classifier.df.mapreduce.partial.Step1Mapper
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:867)
> at
> org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:199)
> at
> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:719)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native
> Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.mahout.classifier.df.mapreduce.partial.Step1Mapper
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native
> Method)
> at
> java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:270)
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:820)
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:865)
> ... 8 more
>
>
> But If I run same command from n2
> [hduser@n2 ~]$ hadoop jar
> mahout-distribution-0.7/mahout-examples-0.7-job.jar
> org.apache.mahout.classifier.df.mapreduce.BuildForest
> -Dmapred.max.split.size=1874231 -p -d testdata/bal_ee_2009.csv
> -ds testdata/bal_ee_2009.csv.info
> <http://bal_ee_2009.csv.info> -sl 10 -o bal_ee_2009_out -t 1
>
> 13/07/11 15:41:20 INFO mapreduce.BuildForest: Partial Mapred
> implementation
> 13/07/11 15:41:20 INFO mapreduce.BuildForest: Building the
> forest...
> 13/07/11 15:41:21 INFO input.FileInputFormat: Total input
> paths to process : 1
> 13/07/11 15:41:21 INFO util.NativeCodeLoader: Loaded the
> native-hadoop library
> 13/07/11 15:41:21 WARN snappy.LoadSnappy: Snappy native
> library not loaded
> 13/07/11 15:41:21 INFO mapred.JobClient: Running job:
> job_201307111355_0017
> 13/07/11 15:41:22 INFO mapred.JobClient: map 0% reduce 0%
> 13/07/11 15:41:43 INFO mapred.JobClient: map 7% reduce 0%
> 13/07/11 15:41:46 INFO mapred.JobClient: map 15% reduce 0%
> 13/07/11 15:41:49 INFO mapred.JobClient: map 38% reduce 0%
> 13/07/11 15:41:52 INFO mapred.JobClient: map 46% reduce 0%
> 13/07/11 15:42:01 INFO mapred.JobClient: map 61% reduce 0%
> 13/07/11 15:42:02 INFO mapred.JobClient: map 76% reduce 0%
> 13/07/11 15:42:09 INFO mapred.JobClient: map 84% reduce 0%
> 13/07/11 15:42:13 INFO mapred.JobClient: map 100% reduce 0%
> 13/07/11 15:42:19 INFO mapred.JobClient: Job complete:
> job_201307111355_0017
> 13/07/11 15:42:19 INFO mapred.JobClient: Counters: 20
> 13/07/11 15:42:19 INFO mapred.JobClient: Job Counters
> 13/07/11 15:42:19 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=138911
> 13/07/11 15:42:19 INFO mapred.JobClient: Total time spent
> by all reduces waiting after reserving slots (ms)=0
> 13/07/11 15:42:19 INFO mapred.JobClient: Total time spent
> by all maps waiting after reserving slots (ms)=0
> 13/07/11 15:42:19 INFO mapred.JobClient: Rack-local map
> tasks=6
> 13/07/11 15:42:19 INFO mapred.JobClient: Launched map tasks=13
> 13/07/11 15:42:19 INFO mapred.JobClient: Data-local map
> tasks=7
> 13/07/11 15:42:19 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
> 13/07/11 15:42:19 INFO mapred.JobClient: File Output Format
> Counters
> 13/07/11 15:42:19 INFO mapred.JobClient: Bytes Written=184956
> 13/07/11 15:42:19 INFO mapred.JobClient: FileSystemCounters
> 13/07/11 15:42:19 INFO mapred.JobClient: HDFS_BYTES_READ=23444612
> 13/07/11 15:42:19 INFO mapred.JobClient: FILE_BYTES_WRITTEN=296026
> 13/07/11 15:42:19 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=184956
> 13/07/11 15:42:19 INFO mapred.JobClient: File Input Format
> Counters
> 13/07/11 15:42:19 INFO mapred.JobClient: Bytes Read=22952913
> 13/07/11 15:42:19 INFO mapred.JobClient: Map-Reduce Framework
> 13/07/11 15:42:19 INFO mapred.JobClient: Map input
> records=204091
> 13/07/11 15:42:19 INFO mapred.JobClient: Physical memory
> (bytes) snapshot=1370255360
> 13/07/11 15:42:19 INFO mapred.JobClient: Spilled Records=0
> 13/07/11 15:42:19 INFO mapred.JobClient: CPU time spent
> (ms)=27990
> 13/07/11 15:42:19 INFO mapred.JobClient: Total committed
> heap usage (bytes)=1292500992
> 13/07/11 15:42:19 INFO mapred.JobClient: Virtual memory
> (bytes) snapshot=37360082944
> 13/07/11 15:42:19 INFO mapred.JobClient: Map output records=1
> 13/07/11 15:42:19 INFO mapred.JobClient: SPLIT_RAW_BYTES=1573
> 13/07/11 15:42:20 INFO common.HadoopUtil: Deleting
> hdfs://master:54310/user/hduser/bal_ee_2009_out
> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Build Time: 0h
> 0m 59s 386
> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Forest num
> Nodes: 10698
> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Forest mean num
> Nodes: 10698
> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Forest mean max
> Depth: 16
> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Storing the
> forest in: bal_ee_2009_out/forest.seq
>
> Both (n1 and n2) are used and from web console I can see that
> there are no errors.
>
> Is there any explanations why I am getting errors when I run
> command from master?
>
>
> --
> Regards, Margus (Margusja) Roo
> +372 51 48 780
> http://margus.roo.ee
> skype: margusja
> -----BEGIN PUBLIC KEY-----
> MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCvbeg7LwEC2SCpAEewwpC3ajxE
> 5ZsRMCB77L8bae9G7TslgLkoIzo9yOjPdx2NN6DllKbV65UjTay43uUDyql9g3tl
> RhiJIcoAExkSTykWqAIPR88LfilLy1JlQ+0RD8OXiWOVVQfhOHpQ0R/jcAkM2lZa
> BjM8j36yJvoBVsfOHQIDAQAB
> -----END PUBLIC KEY-----
>
Re: Task failure in slave node
Posted by Margusja <ma...@roo.ee>.
Than you, it resolved the problem.
Funny, I don't remember that I copied mahout libs to n1 hadoop but there
they are.
Tervitades, Margus (Margusja) Roo
+372 51 48 780
http://margus.roo.ee
skype: margusja
-----BEGIN PUBLIC KEY-----
MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCvbeg7LwEC2SCpAEewwpC3ajxE
5ZsRMCB77L8bae9G7TslgLkoIzo9yOjPdx2NN6DllKbV65UjTay43uUDyql9g3tl
RhiJIcoAExkSTykWqAIPR88LfilLy1JlQ+0RD8OXiWOVVQfhOHpQ0R/jcAkM2lZa
BjM8j36yJvoBVsfOHQIDAQAB
-----END PUBLIC KEY-----
On 7/11/13 4:41 PM, Azuryy Yu wrote:
>
> sorry for typo,
>
> mahout, not mahou. sent from mobile
>
> On Jul 11, 2013 9:40 PM, "Azuryy Yu" <azuryyyu@gmail.com
> <ma...@gmail.com>> wrote:
>
> hi,
>
> put all mahou jars under hadoop_home/lib, then restart cluster.
>
> On Jul 11, 2013 8:45 PM, "Margusja" <margus@roo.ee
> <ma...@roo.ee>> wrote:
>
> Hi
>
> I have tow nodes:
> n1 (master, salve) and n2 (slave)
>
> after set up I ran wordcount example and it worked fine:
> [hduser@n1 ~]$ hadoop jar
> /usr/local/hadoop/hadoop-examples-1.0.4.jar wordcount
> /user/hduser/gutenberg /user/hduser/gutenberg-output
> 13/07/11 15:30:44 INFO input.FileInputFormat: Total input
> paths to process : 7
> 13/07/11 15:30:44 INFO util.NativeCodeLoader: Loaded the
> native-hadoop library
> 13/07/11 15:30:44 WARN snappy.LoadSnappy: Snappy native
> library not loaded
> 13/07/11 15:30:44 INFO mapred.JobClient: Running job:
> job_201307111355_0015
> 13/07/11 15:30:45 INFO mapred.JobClient: map 0% reduce 0%
> 13/07/11 15:31:03 INFO mapred.JobClient: map 42% reduce 0%
> 13/07/11 15:31:06 INFO mapred.JobClient: map 57% reduce 0%
> 13/07/11 15:31:09 INFO mapred.JobClient: map 71% reduce 0%
> 13/07/11 15:31:15 INFO mapred.JobClient: map 100% reduce 0%
> 13/07/11 15:31:18 INFO mapred.JobClient: map 100% reduce 23%
> 13/07/11 15:31:27 INFO mapred.JobClient: map 100% reduce 100%
> 13/07/11 15:31:32 INFO mapred.JobClient: Job complete:
> job_201307111355_0015
> 13/07/11 15:31:32 INFO mapred.JobClient: Counters: 30
> 13/07/11 15:31:32 INFO mapred.JobClient: Job Counters
> 13/07/11 15:31:32 INFO mapred.JobClient: Launched reduce
> tasks=1
> 13/07/11 15:31:32 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=67576
> 13/07/11 15:31:32 INFO mapred.JobClient: Total time spent
> by all reduces waiting after reserving slots (ms)=0
> 13/07/11 15:31:32 INFO mapred.JobClient: Total time spent
> by all maps waiting after reserving slots (ms)=0
> 13/07/11 15:31:32 INFO mapred.JobClient: Rack-local map
> tasks=3
> 13/07/11 15:31:32 INFO mapred.JobClient: Launched map tasks=7
> 13/07/11 15:31:32 INFO mapred.JobClient: Data-local map
> tasks=4
> 13/07/11 15:31:32 INFO mapred.JobClient:
> SLOTS_MILLIS_REDUCES=21992
> 13/07/11 15:31:32 INFO mapred.JobClient: File Output Format
> Counters
> 13/07/11 15:31:32 INFO mapred.JobClient: Bytes Written=1412505
> 13/07/11 15:31:32 INFO mapred.JobClient: FileSystemCounters
> 13/07/11 15:31:32 INFO mapred.JobClient: FILE_BYTES_READ=5414195
> 13/07/11 15:31:32 INFO mapred.JobClient: HDFS_BYTES_READ=6950820
> 13/07/11 15:31:32 INFO mapred.JobClient:
> FILE_BYTES_WRITTEN=8744993
> 13/07/11 15:31:32 INFO mapred.JobClient:
> HDFS_BYTES_WRITTEN=1412505
> 13/07/11 15:31:32 INFO mapred.JobClient: File Input Format
> Counters
> 13/07/11 15:31:32 INFO mapred.JobClient: Bytes Read=6950001
> 13/07/11 15:31:32 INFO mapred.JobClient: Map-Reduce Framework
> 13/07/11 15:31:32 INFO mapred.JobClient: Map output
> materialized bytes=3157469
> 13/07/11 15:31:32 INFO mapred.JobClient: Map input
> records=137146
> 13/07/11 15:31:32 INFO mapred.JobClient: Reduce shuffle
> bytes=2904836
> 13/07/11 15:31:32 INFO mapred.JobClient: Spilled
> Records=594764
> 13/07/11 15:31:32 INFO mapred.JobClient: Map output
> bytes=11435849
> 13/07/11 15:31:32 INFO mapred.JobClient: Total committed
> heap usage (bytes)=1128136704
> 13/07/11 15:31:32 INFO mapred.JobClient: CPU time spent
> (ms)=18230
> 13/07/11 15:31:32 INFO mapred.JobClient: Combine input
> records=1174991
> 13/07/11 15:31:32 INFO mapred.JobClient: SPLIT_RAW_BYTES=819
> 13/07/11 15:31:32 INFO mapred.JobClient: Reduce input
> records=218990
> 13/07/11 15:31:32 INFO mapred.JobClient: Reduce input
> groups=128513
> 13/07/11 15:31:32 INFO mapred.JobClient: Combine output
> records=218990
> 13/07/11 15:31:32 INFO mapred.JobClient: Physical memory
> (bytes) snapshot=1179656192
> 13/07/11 15:31:32 INFO mapred.JobClient: Reduce output
> records=128513
> 13/07/11 15:31:32 INFO mapred.JobClient: Virtual memory
> (bytes) snapshot=22992117760
> 13/07/11 15:31:32 INFO mapred.JobClient: Map output
> records=1174991
>
> from web interface (http://n1:50030/) I saw that both (n1 and
> n2 ) were used without any errors.
>
> Problems appear if I try to use following commands in master (n1):
>
> [hduser@n1 ~]$hadoop jar
> mahout-distribution-0.7/mahout-examples-0.7-job.jar
> org.apache.mahout.classifier.df.mapreduce.BuildForest
> -Dmapred.max.split.size=1874231 -p -d testdata/bal_ee_2009.csv
> -ds testdata/bal_ee_2009.csv.info
> <http://bal_ee_2009.csv.info> -sl 10 -o bal_ee_2009_out -t 1
>
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [file:/usr/local/hadoop-1.0.4/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/usr/local/hadoop-1.0.4/lib/slf4j-log4j12-1.4.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings
> for an explanation.
> 13/07/11 15:36:50 INFO mapreduce.BuildForest: Partial Mapred
> implementation
> 13/07/11 15:36:50 INFO mapreduce.BuildForest: Building the
> forest...
> 13/07/11 15:36:50 WARN mapred.JobClient: No job jar file set.
> User classes may not be found. See JobConf(Class) or
> JobConf#setJar(String).
> 13/07/11 15:36:50 INFO input.FileInputFormat: Total input
> paths to process : 1
> 13/07/11 15:36:50 INFO util.NativeCodeLoader: Loaded the
> native-hadoop library
> 13/07/11 15:36:50 WARN snappy.LoadSnappy: Snappy native
> library not loaded
> 13/07/11 15:36:50 INFO mapred.JobClient: Running job:
> job_201307111355_0016
> 13/07/11 15:36:51 INFO mapred.JobClient: map 0% reduce 0%
> 13/07/11 15:37:09 INFO mapred.JobClient: map 7% reduce 0%
> 13/07/11 15:37:11 INFO mapred.JobClient: Task Id :
> attempt_201307111355_0016_m_000002_0, Status : FAILED
> java.lang.RuntimeException: java.lang.ClassNotFoundException:
> org.apache.mahout.classifier.df.mapreduce.partial.Step1Mapper
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:867)
> at
> org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:199)
> at
> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:719)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native
> Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.mahout.classifier.df.mapreduce.partial.Step1Mapper
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native
> Method)
> at
> java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:270)
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:820)
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:865)
> ... 8 more
>
> ...
> ...
> ...
>
> 13/07/11 15:37:37 INFO mapred.JobClient: map 61% reduce 0%
> 13/07/11 15:37:46 INFO mapred.JobClient: map 76% reduce 0%
> 13/07/11 15:37:55 INFO mapred.JobClient: map 92% reduce 0%
> 13/07/11 15:38:01 INFO mapred.JobClient: map 100% reduce 0%
> 13/07/11 15:38:06 INFO mapred.JobClient: Job complete:
> job_201307111355_0016
> 13/07/11 15:38:06 INFO mapred.JobClient: Counters: 20
> 13/07/11 15:38:06 INFO mapred.JobClient: Job Counters
> 13/07/11 15:38:06 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=184724
> 13/07/11 15:38:06 INFO mapred.JobClient: Total time spent
> by all reduces waiting after reserving slots (ms)=0
> 13/07/11 15:38:06 INFO mapred.JobClient: Total time spent
> by all maps waiting after reserving slots (ms)=0
> 13/07/11 15:38:06 INFO mapred.JobClient: Rack-local map
> tasks=13
> 13/07/11 15:38:06 INFO mapred.JobClient: Launched map tasks=22
> 13/07/11 15:38:06 INFO mapred.JobClient: Data-local map
> tasks=9
> 13/07/11 15:38:06 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
> 13/07/11 15:38:06 INFO mapred.JobClient: File Output Format
> Counters
> 13/07/11 15:38:06 INFO mapred.JobClient: Bytes Written=184520
> 13/07/11 15:38:06 INFO mapred.JobClient: FileSystemCounters
> 13/07/11 15:38:06 INFO mapred.JobClient: HDFS_BYTES_READ=23444612
> 13/07/11 15:38:06 INFO mapred.JobClient: FILE_BYTES_WRITTEN=294037
> 13/07/11 15:38:06 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=184520
> 13/07/11 15:38:06 INFO mapred.JobClient: File Input Format
> Counters
> 13/07/11 15:38:06 INFO mapred.JobClient: Bytes Read=22952913
> 13/07/11 15:38:06 INFO mapred.JobClient: Map-Reduce Framework
> 13/07/11 15:38:06 INFO mapred.JobClient: Map input
> records=204091
> 13/07/11 15:38:06 INFO mapred.JobClient: Physical memory
> (bytes) snapshot=1402363904
> 13/07/11 15:38:06 INFO mapred.JobClient: Spilled Records=0
> 13/07/11 15:38:06 INFO mapred.JobClient: CPU time spent
> (ms)=26640
> 13/07/11 15:38:06 INFO mapred.JobClient: Total committed
> heap usage (bytes)=1333985280
> 13/07/11 15:38:06 INFO mapred.JobClient: Virtual memory
> (bytes) snapshot=37349126144
> 13/07/11 15:38:06 INFO mapred.JobClient: Map output records=1
> 13/07/11 15:38:06 INFO mapred.JobClient: SPLIT_RAW_BYTES=1573
> 13/07/11 15:38:07 INFO common.HadoopUtil: Deleting
> hdfs://master:54310/user/hduser/bal_ee_2009_out
> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Build Time: 0h
> 1m 16s 977
> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Forest num
> Nodes: 10700
> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Forest mean num
> Nodes: 10700
> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Forest mean max
> Depth: 17
> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Storing the
> forest in: bal_ee_2009_out/forest.seq
>
> From web console I can see that n1 tasks are fine but n2 tasks
> failed:
>
> java.lang.RuntimeException: java.lang.ClassNotFoundException:
> org.apache.mahout.classifier.df.mapreduce.partial.Step1Mapper
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:867)
> at
> org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:199)
> at
> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:719)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native
> Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.mahout.classifier.df.mapreduce.partial.Step1Mapper
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native
> Method)
> at
> java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:270)
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:820)
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:865)
> ... 8 more
>
>
> But If I run same command from n2
> [hduser@n2 ~]$ hadoop jar
> mahout-distribution-0.7/mahout-examples-0.7-job.jar
> org.apache.mahout.classifier.df.mapreduce.BuildForest
> -Dmapred.max.split.size=1874231 -p -d testdata/bal_ee_2009.csv
> -ds testdata/bal_ee_2009.csv.info
> <http://bal_ee_2009.csv.info> -sl 10 -o bal_ee_2009_out -t 1
>
> 13/07/11 15:41:20 INFO mapreduce.BuildForest: Partial Mapred
> implementation
> 13/07/11 15:41:20 INFO mapreduce.BuildForest: Building the
> forest...
> 13/07/11 15:41:21 INFO input.FileInputFormat: Total input
> paths to process : 1
> 13/07/11 15:41:21 INFO util.NativeCodeLoader: Loaded the
> native-hadoop library
> 13/07/11 15:41:21 WARN snappy.LoadSnappy: Snappy native
> library not loaded
> 13/07/11 15:41:21 INFO mapred.JobClient: Running job:
> job_201307111355_0017
> 13/07/11 15:41:22 INFO mapred.JobClient: map 0% reduce 0%
> 13/07/11 15:41:43 INFO mapred.JobClient: map 7% reduce 0%
> 13/07/11 15:41:46 INFO mapred.JobClient: map 15% reduce 0%
> 13/07/11 15:41:49 INFO mapred.JobClient: map 38% reduce 0%
> 13/07/11 15:41:52 INFO mapred.JobClient: map 46% reduce 0%
> 13/07/11 15:42:01 INFO mapred.JobClient: map 61% reduce 0%
> 13/07/11 15:42:02 INFO mapred.JobClient: map 76% reduce 0%
> 13/07/11 15:42:09 INFO mapred.JobClient: map 84% reduce 0%
> 13/07/11 15:42:13 INFO mapred.JobClient: map 100% reduce 0%
> 13/07/11 15:42:19 INFO mapred.JobClient: Job complete:
> job_201307111355_0017
> 13/07/11 15:42:19 INFO mapred.JobClient: Counters: 20
> 13/07/11 15:42:19 INFO mapred.JobClient: Job Counters
> 13/07/11 15:42:19 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=138911
> 13/07/11 15:42:19 INFO mapred.JobClient: Total time spent
> by all reduces waiting after reserving slots (ms)=0
> 13/07/11 15:42:19 INFO mapred.JobClient: Total time spent
> by all maps waiting after reserving slots (ms)=0
> 13/07/11 15:42:19 INFO mapred.JobClient: Rack-local map
> tasks=6
> 13/07/11 15:42:19 INFO mapred.JobClient: Launched map tasks=13
> 13/07/11 15:42:19 INFO mapred.JobClient: Data-local map
> tasks=7
> 13/07/11 15:42:19 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
> 13/07/11 15:42:19 INFO mapred.JobClient: File Output Format
> Counters
> 13/07/11 15:42:19 INFO mapred.JobClient: Bytes Written=184956
> 13/07/11 15:42:19 INFO mapred.JobClient: FileSystemCounters
> 13/07/11 15:42:19 INFO mapred.JobClient: HDFS_BYTES_READ=23444612
> 13/07/11 15:42:19 INFO mapred.JobClient: FILE_BYTES_WRITTEN=296026
> 13/07/11 15:42:19 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=184956
> 13/07/11 15:42:19 INFO mapred.JobClient: File Input Format
> Counters
> 13/07/11 15:42:19 INFO mapred.JobClient: Bytes Read=22952913
> 13/07/11 15:42:19 INFO mapred.JobClient: Map-Reduce Framework
> 13/07/11 15:42:19 INFO mapred.JobClient: Map input
> records=204091
> 13/07/11 15:42:19 INFO mapred.JobClient: Physical memory
> (bytes) snapshot=1370255360
> 13/07/11 15:42:19 INFO mapred.JobClient: Spilled Records=0
> 13/07/11 15:42:19 INFO mapred.JobClient: CPU time spent
> (ms)=27990
> 13/07/11 15:42:19 INFO mapred.JobClient: Total committed
> heap usage (bytes)=1292500992
> 13/07/11 15:42:19 INFO mapred.JobClient: Virtual memory
> (bytes) snapshot=37360082944
> 13/07/11 15:42:19 INFO mapred.JobClient: Map output records=1
> 13/07/11 15:42:19 INFO mapred.JobClient: SPLIT_RAW_BYTES=1573
> 13/07/11 15:42:20 INFO common.HadoopUtil: Deleting
> hdfs://master:54310/user/hduser/bal_ee_2009_out
> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Build Time: 0h
> 0m 59s 386
> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Forest num
> Nodes: 10698
> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Forest mean num
> Nodes: 10698
> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Forest mean max
> Depth: 16
> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Storing the
> forest in: bal_ee_2009_out/forest.seq
>
> Both (n1 and n2) are used and from web console I can see that
> there are no errors.
>
> Is there any explanations why I am getting errors when I run
> command from master?
>
>
> --
> Regards, Margus (Margusja) Roo
> +372 51 48 780
> http://margus.roo.ee
> skype: margusja
> -----BEGIN PUBLIC KEY-----
> MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCvbeg7LwEC2SCpAEewwpC3ajxE
> 5ZsRMCB77L8bae9G7TslgLkoIzo9yOjPdx2NN6DllKbV65UjTay43uUDyql9g3tl
> RhiJIcoAExkSTykWqAIPR88LfilLy1JlQ+0RD8OXiWOVVQfhOHpQ0R/jcAkM2lZa
> BjM8j36yJvoBVsfOHQIDAQAB
> -----END PUBLIC KEY-----
>
Re: Task failure in slave node
Posted by Azuryy Yu <az...@gmail.com>.
sorry for typo,
mahout, not mahou. sent from mobile
On Jul 11, 2013 9:40 PM, "Azuryy Yu" <az...@gmail.com> wrote:
> hi,
>
> put all mahou jars under hadoop_home/lib, then restart cluster.
> On Jul 11, 2013 8:45 PM, "Margusja" <ma...@roo.ee> wrote:
>
>> Hi
>>
>> I have tow nodes:
>> n1 (master, salve) and n2 (slave)
>>
>> after set up I ran wordcount example and it worked fine:
>> [hduser@n1 ~]$ hadoop jar /usr/local/hadoop/hadoop-**examples-1.0.4.jar
>> wordcount /user/hduser/gutenberg /user/hduser/gutenberg-output
>> 13/07/11 15:30:44 INFO input.FileInputFormat: Total input paths to
>> process : 7
>> 13/07/11 15:30:44 INFO util.NativeCodeLoader: Loaded the native-hadoop
>> library
>> 13/07/11 15:30:44 WARN snappy.LoadSnappy: Snappy native library not loaded
>> 13/07/11 15:30:44 INFO mapred.JobClient: Running job:
>> job_201307111355_0015
>> 13/07/11 15:30:45 INFO mapred.JobClient: map 0% reduce 0%
>> 13/07/11 15:31:03 INFO mapred.JobClient: map 42% reduce 0%
>> 13/07/11 15:31:06 INFO mapred.JobClient: map 57% reduce 0%
>> 13/07/11 15:31:09 INFO mapred.JobClient: map 71% reduce 0%
>> 13/07/11 15:31:15 INFO mapred.JobClient: map 100% reduce 0%
>> 13/07/11 15:31:18 INFO mapred.JobClient: map 100% reduce 23%
>> 13/07/11 15:31:27 INFO mapred.JobClient: map 100% reduce 100%
>> 13/07/11 15:31:32 INFO mapred.JobClient: Job complete:
>> job_201307111355_0015
>> 13/07/11 15:31:32 INFO mapred.JobClient: Counters: 30
>> 13/07/11 15:31:32 INFO mapred.JobClient: Job Counters
>> 13/07/11 15:31:32 INFO mapred.JobClient: Launched reduce tasks=1
>> 13/07/11 15:31:32 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=67576
>> 13/07/11 15:31:32 INFO mapred.JobClient: Total time spent by all
>> reduces waiting after reserving slots (ms)=0
>> 13/07/11 15:31:32 INFO mapred.JobClient: Total time spent by all maps
>> waiting after reserving slots (ms)=0
>> 13/07/11 15:31:32 INFO mapred.JobClient: Rack-local map tasks=3
>> 13/07/11 15:31:32 INFO mapred.JobClient: Launched map tasks=7
>> 13/07/11 15:31:32 INFO mapred.JobClient: Data-local map tasks=4
>> 13/07/11 15:31:32 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=21992
>> 13/07/11 15:31:32 INFO mapred.JobClient: File Output Format Counters
>> 13/07/11 15:31:32 INFO mapred.JobClient: Bytes Written=1412505
>> 13/07/11 15:31:32 INFO mapred.JobClient: FileSystemCounters
>> 13/07/11 15:31:32 INFO mapred.JobClient: FILE_BYTES_READ=5414195
>> 13/07/11 15:31:32 INFO mapred.JobClient: HDFS_BYTES_READ=6950820
>> 13/07/11 15:31:32 INFO mapred.JobClient: FILE_BYTES_WRITTEN=8744993
>> 13/07/11 15:31:32 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=1412505
>> 13/07/11 15:31:32 INFO mapred.JobClient: File Input Format Counters
>> 13/07/11 15:31:32 INFO mapred.JobClient: Bytes Read=6950001
>> 13/07/11 15:31:32 INFO mapred.JobClient: Map-Reduce Framework
>> 13/07/11 15:31:32 INFO mapred.JobClient: Map output materialized
>> bytes=3157469
>> 13/07/11 15:31:32 INFO mapred.JobClient: Map input records=137146
>> 13/07/11 15:31:32 INFO mapred.JobClient: Reduce shuffle bytes=2904836
>> 13/07/11 15:31:32 INFO mapred.JobClient: Spilled Records=594764
>> 13/07/11 15:31:32 INFO mapred.JobClient: Map output bytes=11435849
>> 13/07/11 15:31:32 INFO mapred.JobClient: Total committed heap usage
>> (bytes)=1128136704
>> 13/07/11 15:31:32 INFO mapred.JobClient: CPU time spent (ms)=18230
>> 13/07/11 15:31:32 INFO mapred.JobClient: Combine input records=1174991
>> 13/07/11 15:31:32 INFO mapred.JobClient: SPLIT_RAW_BYTES=819
>> 13/07/11 15:31:32 INFO mapred.JobClient: Reduce input records=218990
>> 13/07/11 15:31:32 INFO mapred.JobClient: Reduce input groups=128513
>> 13/07/11 15:31:32 INFO mapred.JobClient: Combine output records=218990
>> 13/07/11 15:31:32 INFO mapred.JobClient: Physical memory (bytes)
>> snapshot=1179656192
>> 13/07/11 15:31:32 INFO mapred.JobClient: Reduce output records=128513
>> 13/07/11 15:31:32 INFO mapred.JobClient: Virtual memory (bytes)
>> snapshot=22992117760
>> 13/07/11 15:31:32 INFO mapred.JobClient: Map output records=1174991
>>
>> from web interface (http://n1:50030/) I saw that both (n1 and n2 ) were
>> used without any errors.
>>
>> Problems appear if I try to use following commands in master (n1):
>>
>> [hduser@n1 ~]$hadoop jar mahout-distribution-0.7/**mahout-examples-0.7-job.jar
>> org.apache.mahout.classifier.**df.mapreduce.BuildForest
>> -Dmapred.max.split.size=**1874231 -p -d testdata/bal_ee_2009.csv -ds
>> testdata/bal_ee_2009.csv.info -sl 10 -o bal_ee_2009_out -t 1
>>
>> SLF4J: Class path contains multiple SLF4J bindings.
>> SLF4J: Found binding in [file:/usr/local/hadoop-1.0.4/**org/slf4j/impl/**
>> StaticLoggerBinder.class]
>> SLF4J: Found binding in [jar:file:/usr/local/hadoop-1.**
>> 0.4/lib/slf4j-log4j12-1.4.3.**jar!/org/slf4j/impl/**
>> StaticLoggerBinder.class]
>> SLF4J: See http://www.slf4j.org/codes.**html#multiple_bindings<http://www.slf4j.org/codes.html#multiple_bindings>for an explanation.
>> 13/07/11 15:36:50 INFO mapreduce.BuildForest: Partial Mapred
>> implementation
>> 13/07/11 15:36:50 INFO mapreduce.BuildForest: Building the forest...
>> 13/07/11 15:36:50 WARN mapred.JobClient: No job jar file set. User
>> classes may not be found. See JobConf(Class) or JobConf#setJar(String).
>> 13/07/11 15:36:50 INFO input.FileInputFormat: Total input paths to
>> process : 1
>> 13/07/11 15:36:50 INFO util.NativeCodeLoader: Loaded the native-hadoop
>> library
>> 13/07/11 15:36:50 WARN snappy.LoadSnappy: Snappy native library not loaded
>> 13/07/11 15:36:50 INFO mapred.JobClient: Running job:
>> job_201307111355_0016
>> 13/07/11 15:36:51 INFO mapred.JobClient: map 0% reduce 0%
>> 13/07/11 15:37:09 INFO mapred.JobClient: map 7% reduce 0%
>> 13/07/11 15:37:11 INFO mapred.JobClient: Task Id :
>> attempt_201307111355_0016_m_**000002_0, Status : FAILED
>> java.lang.RuntimeException: java.lang.**ClassNotFoundException:
>> org.apache.mahout.classifier.**df.mapreduce.partial.**Step1Mapper
>> at org.apache.hadoop.conf.**Configuration.getClass(**
>> Configuration.java:867)
>> at org.apache.hadoop.mapreduce.**JobContext.getMapperClass(**
>> JobContext.java:199)
>> at org.apache.hadoop.mapred.**MapTask.runNewMapper(MapTask.**
>> java:719)
>> at org.apache.hadoop.mapred.**MapTask.run(MapTask.java:370)
>> at org.apache.hadoop.mapred.**Child$4.run(Child.java:255)
>> at java.security.**AccessController.doPrivileged(**Native Method)
>> at javax.security.auth.Subject.**doAs(Subject.java:415)
>> at org.apache.hadoop.security.**UserGroupInformation.doAs(**
>> UserGroupInformation.java:**1121)
>> at org.apache.hadoop.mapred.**Child.main(Child.java:249)
>> Caused by: java.lang.**ClassNotFoundException:
>> org.apache.mahout.classifier.**df.mapreduce.partial.**Step1Mapper
>> at java.net.URLClassLoader$1.run(**URLClassLoader.java:366)
>> at java.net.URLClassLoader$1.run(**URLClassLoader.java:355)
>> at java.security.**AccessController.doPrivileged(**Native Method)
>> at java.net.URLClassLoader.**findClass(URLClassLoader.java:**354)
>> at java.lang.ClassLoader.**loadClass(ClassLoader.java:**424)
>> at sun.misc.Launcher$**AppClassLoader.loadClass(**
>> Launcher.java:308)
>> at java.lang.ClassLoader.**loadClass(ClassLoader.java:**357)
>> at java.lang.Class.forName0(**Native Method)
>> at java.lang.Class.forName(Class.**java:270)
>> at org.apache.hadoop.conf.**Configuration.getClassByName(**
>> Configuration.java:820)
>> at org.apache.hadoop.conf.**Configuration.getClass(**
>> Configuration.java:865)
>> ... 8 more
>>
>> ...
>> ...
>> ...
>>
>> 13/07/11 15:37:37 INFO mapred.JobClient: map 61% reduce 0%
>> 13/07/11 15:37:46 INFO mapred.JobClient: map 76% reduce 0%
>> 13/07/11 15:37:55 INFO mapred.JobClient: map 92% reduce 0%
>> 13/07/11 15:38:01 INFO mapred.JobClient: map 100% reduce 0%
>> 13/07/11 15:38:06 INFO mapred.JobClient: Job complete:
>> job_201307111355_0016
>> 13/07/11 15:38:06 INFO mapred.JobClient: Counters: 20
>> 13/07/11 15:38:06 INFO mapred.JobClient: Job Counters
>> 13/07/11 15:38:06 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=184724
>> 13/07/11 15:38:06 INFO mapred.JobClient: Total time spent by all
>> reduces waiting after reserving slots (ms)=0
>> 13/07/11 15:38:06 INFO mapred.JobClient: Total time spent by all maps
>> waiting after reserving slots (ms)=0
>> 13/07/11 15:38:06 INFO mapred.JobClient: Rack-local map tasks=13
>> 13/07/11 15:38:06 INFO mapred.JobClient: Launched map tasks=22
>> 13/07/11 15:38:06 INFO mapred.JobClient: Data-local map tasks=9
>> 13/07/11 15:38:06 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
>> 13/07/11 15:38:06 INFO mapred.JobClient: File Output Format Counters
>> 13/07/11 15:38:06 INFO mapred.JobClient: Bytes Written=184520
>> 13/07/11 15:38:06 INFO mapred.JobClient: FileSystemCounters
>> 13/07/11 15:38:06 INFO mapred.JobClient: HDFS_BYTES_READ=23444612
>> 13/07/11 15:38:06 INFO mapred.JobClient: FILE_BYTES_WRITTEN=294037
>> 13/07/11 15:38:06 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=184520
>> 13/07/11 15:38:06 INFO mapred.JobClient: File Input Format Counters
>> 13/07/11 15:38:06 INFO mapred.JobClient: Bytes Read=22952913
>> 13/07/11 15:38:06 INFO mapred.JobClient: Map-Reduce Framework
>> 13/07/11 15:38:06 INFO mapred.JobClient: Map input records=204091
>> 13/07/11 15:38:06 INFO mapred.JobClient: Physical memory (bytes)
>> snapshot=1402363904
>> 13/07/11 15:38:06 INFO mapred.JobClient: Spilled Records=0
>> 13/07/11 15:38:06 INFO mapred.JobClient: CPU time spent (ms)=26640
>> 13/07/11 15:38:06 INFO mapred.JobClient: Total committed heap usage
>> (bytes)=1333985280
>> 13/07/11 15:38:06 INFO mapred.JobClient: Virtual memory (bytes)
>> snapshot=37349126144
>> 13/07/11 15:38:06 INFO mapred.JobClient: Map output records=1
>> 13/07/11 15:38:06 INFO mapred.JobClient: SPLIT_RAW_BYTES=1573
>> 13/07/11 15:38:07 INFO common.HadoopUtil: Deleting
>> hdfs://master:54310/user/**hduser/bal_ee_2009_out
>> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Build Time: 0h 1m 16s 977
>> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Forest num Nodes: 10700
>> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Forest mean num Nodes: 10700
>> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Forest mean max Depth: 17
>> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Storing the forest in:
>> bal_ee_2009_out/forest.seq
>>
>> From web console I can see that n1 tasks are fine but n2 tasks failed:
>>
>> java.lang.RuntimeException: java.lang.**ClassNotFoundException:
>> org.apache.mahout.classifier.**df.mapreduce.partial.**Step1Mapper
>> at org.apache.hadoop.conf.**Configuration.getClass(**
>> Configuration.java:867)
>> at org.apache.hadoop.mapreduce.**JobContext.getMapperClass(**
>> JobContext.java:199)
>> at org.apache.hadoop.mapred.**MapTask.runNewMapper(MapTask.**
>> java:719)
>> at org.apache.hadoop.mapred.**MapTask.run(MapTask.java:370)
>> at org.apache.hadoop.mapred.**Child$4.run(Child.java:255)
>> at java.security.**AccessController.doPrivileged(**Native Method)
>> at javax.security.auth.Subject.**doAs(Subject.java:415)
>> at org.apache.hadoop.security.**UserGroupInformation.doAs(**
>> UserGroupInformation.java:**1121)
>> at org.apache.hadoop.mapred.**Child.main(Child.java:249)
>> Caused by: java.lang.**ClassNotFoundException:
>> org.apache.mahout.classifier.**df.mapreduce.partial.**Step1Mapper
>> at java.net.URLClassLoader$1.run(**URLClassLoader.java:366)
>> at java.net.URLClassLoader$1.run(**URLClassLoader.java:355)
>> at java.security.**AccessController.doPrivileged(**Native Method)
>> at java.net.URLClassLoader.**findClass(URLClassLoader.java:**354)
>> at java.lang.ClassLoader.**loadClass(ClassLoader.java:**424)
>> at sun.misc.Launcher$**AppClassLoader.loadClass(**
>> Launcher.java:308)
>> at java.lang.ClassLoader.**loadClass(ClassLoader.java:**357)
>> at java.lang.Class.forName0(**Native Method)
>> at java.lang.Class.forName(Class.**java:270)
>> at org.apache.hadoop.conf.**Configuration.getClassByName(**
>> Configuration.java:820)
>> at org.apache.hadoop.conf.**Configuration.getClass(**
>> Configuration.java:865)
>> ... 8 more
>>
>>
>> But If I run same command from n2
>> [hduser@n2 ~]$ hadoop jar mahout-distribution-0.7/**mahout-examples-0.7-job.jar
>> org.apache.mahout.classifier.**df.mapreduce.BuildForest
>> -Dmapred.max.split.size=**1874231 -p -d testdata/bal_ee_2009.csv -ds
>> testdata/bal_ee_2009.csv.info -sl 10 -o bal_ee_2009_out -t 1
>>
>> 13/07/11 15:41:20 INFO mapreduce.BuildForest: Partial Mapred
>> implementation
>> 13/07/11 15:41:20 INFO mapreduce.BuildForest: Building the forest...
>> 13/07/11 15:41:21 INFO input.FileInputFormat: Total input paths to
>> process : 1
>> 13/07/11 15:41:21 INFO util.NativeCodeLoader: Loaded the native-hadoop
>> library
>> 13/07/11 15:41:21 WARN snappy.LoadSnappy: Snappy native library not loaded
>> 13/07/11 15:41:21 INFO mapred.JobClient: Running job:
>> job_201307111355_0017
>> 13/07/11 15:41:22 INFO mapred.JobClient: map 0% reduce 0%
>> 13/07/11 15:41:43 INFO mapred.JobClient: map 7% reduce 0%
>> 13/07/11 15:41:46 INFO mapred.JobClient: map 15% reduce 0%
>> 13/07/11 15:41:49 INFO mapred.JobClient: map 38% reduce 0%
>> 13/07/11 15:41:52 INFO mapred.JobClient: map 46% reduce 0%
>> 13/07/11 15:42:01 INFO mapred.JobClient: map 61% reduce 0%
>> 13/07/11 15:42:02 INFO mapred.JobClient: map 76% reduce 0%
>> 13/07/11 15:42:09 INFO mapred.JobClient: map 84% reduce 0%
>> 13/07/11 15:42:13 INFO mapred.JobClient: map 100% reduce 0%
>> 13/07/11 15:42:19 INFO mapred.JobClient: Job complete:
>> job_201307111355_0017
>> 13/07/11 15:42:19 INFO mapred.JobClient: Counters: 20
>> 13/07/11 15:42:19 INFO mapred.JobClient: Job Counters
>> 13/07/11 15:42:19 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=138911
>> 13/07/11 15:42:19 INFO mapred.JobClient: Total time spent by all
>> reduces waiting after reserving slots (ms)=0
>> 13/07/11 15:42:19 INFO mapred.JobClient: Total time spent by all maps
>> waiting after reserving slots (ms)=0
>> 13/07/11 15:42:19 INFO mapred.JobClient: Rack-local map tasks=6
>> 13/07/11 15:42:19 INFO mapred.JobClient: Launched map tasks=13
>> 13/07/11 15:42:19 INFO mapred.JobClient: Data-local map tasks=7
>> 13/07/11 15:42:19 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
>> 13/07/11 15:42:19 INFO mapred.JobClient: File Output Format Counters
>> 13/07/11 15:42:19 INFO mapred.JobClient: Bytes Written=184956
>> 13/07/11 15:42:19 INFO mapred.JobClient: FileSystemCounters
>> 13/07/11 15:42:19 INFO mapred.JobClient: HDFS_BYTES_READ=23444612
>> 13/07/11 15:42:19 INFO mapred.JobClient: FILE_BYTES_WRITTEN=296026
>> 13/07/11 15:42:19 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=184956
>> 13/07/11 15:42:19 INFO mapred.JobClient: File Input Format Counters
>> 13/07/11 15:42:19 INFO mapred.JobClient: Bytes Read=22952913
>> 13/07/11 15:42:19 INFO mapred.JobClient: Map-Reduce Framework
>> 13/07/11 15:42:19 INFO mapred.JobClient: Map input records=204091
>> 13/07/11 15:42:19 INFO mapred.JobClient: Physical memory (bytes)
>> snapshot=1370255360
>> 13/07/11 15:42:19 INFO mapred.JobClient: Spilled Records=0
>> 13/07/11 15:42:19 INFO mapred.JobClient: CPU time spent (ms)=27990
>> 13/07/11 15:42:19 INFO mapred.JobClient: Total committed heap usage
>> (bytes)=1292500992
>> 13/07/11 15:42:19 INFO mapred.JobClient: Virtual memory (bytes)
>> snapshot=37360082944
>> 13/07/11 15:42:19 INFO mapred.JobClient: Map output records=1
>> 13/07/11 15:42:19 INFO mapred.JobClient: SPLIT_RAW_BYTES=1573
>> 13/07/11 15:42:20 INFO common.HadoopUtil: Deleting
>> hdfs://master:54310/user/**hduser/bal_ee_2009_out
>> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Build Time: 0h 0m 59s 386
>> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Forest num Nodes: 10698
>> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Forest mean num Nodes: 10698
>> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Forest mean max Depth: 16
>> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Storing the forest in:
>> bal_ee_2009_out/forest.seq
>>
>> Both (n1 and n2) are used and from web console I can see that there are
>> no errors.
>>
>> Is there any explanations why I am getting errors when I run command from
>> master?
>>
>>
>> --
>> Regards, Margus (Margusja) Roo
>> +372 51 48 780
>> http://margus.roo.ee
>> skype: margusja
>> -----BEGIN PUBLIC KEY-----
>> MIGfMA0GCSqGSIb3DQEBAQUAA4GNAD**CBiQKBgQCvbeg7LwEC2SCpAEewwpC3**ajxE
>> 5ZsRMCB77L8bae9G7TslgLkoIzo9yO**jPdx2NN6DllKbV65UjTay43uUDyql9**g3tl
>> RhiJIcoAExkSTykWqAIPR88LfilLy1**JlQ+0RD8OXiWOVVQfhOHpQ0R/**jcAkM2lZa
>> BjM8j36yJvoBVsfOHQIDAQAB
>> -----END PUBLIC KEY-----
>>
>>
Re: Task failure in slave node
Posted by Azuryy Yu <az...@gmail.com>.
sorry for typo,
mahout, not mahou. sent from mobile
On Jul 11, 2013 9:40 PM, "Azuryy Yu" <az...@gmail.com> wrote:
> hi,
>
> put all mahou jars under hadoop_home/lib, then restart cluster.
> On Jul 11, 2013 8:45 PM, "Margusja" <ma...@roo.ee> wrote:
>
>> Hi
>>
>> I have tow nodes:
>> n1 (master, salve) and n2 (slave)
>>
>> after set up I ran wordcount example and it worked fine:
>> [hduser@n1 ~]$ hadoop jar /usr/local/hadoop/hadoop-**examples-1.0.4.jar
>> wordcount /user/hduser/gutenberg /user/hduser/gutenberg-output
>> 13/07/11 15:30:44 INFO input.FileInputFormat: Total input paths to
>> process : 7
>> 13/07/11 15:30:44 INFO util.NativeCodeLoader: Loaded the native-hadoop
>> library
>> 13/07/11 15:30:44 WARN snappy.LoadSnappy: Snappy native library not loaded
>> 13/07/11 15:30:44 INFO mapred.JobClient: Running job:
>> job_201307111355_0015
>> 13/07/11 15:30:45 INFO mapred.JobClient: map 0% reduce 0%
>> 13/07/11 15:31:03 INFO mapred.JobClient: map 42% reduce 0%
>> 13/07/11 15:31:06 INFO mapred.JobClient: map 57% reduce 0%
>> 13/07/11 15:31:09 INFO mapred.JobClient: map 71% reduce 0%
>> 13/07/11 15:31:15 INFO mapred.JobClient: map 100% reduce 0%
>> 13/07/11 15:31:18 INFO mapred.JobClient: map 100% reduce 23%
>> 13/07/11 15:31:27 INFO mapred.JobClient: map 100% reduce 100%
>> 13/07/11 15:31:32 INFO mapred.JobClient: Job complete:
>> job_201307111355_0015
>> 13/07/11 15:31:32 INFO mapred.JobClient: Counters: 30
>> 13/07/11 15:31:32 INFO mapred.JobClient: Job Counters
>> 13/07/11 15:31:32 INFO mapred.JobClient: Launched reduce tasks=1
>> 13/07/11 15:31:32 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=67576
>> 13/07/11 15:31:32 INFO mapred.JobClient: Total time spent by all
>> reduces waiting after reserving slots (ms)=0
>> 13/07/11 15:31:32 INFO mapred.JobClient: Total time spent by all maps
>> waiting after reserving slots (ms)=0
>> 13/07/11 15:31:32 INFO mapred.JobClient: Rack-local map tasks=3
>> 13/07/11 15:31:32 INFO mapred.JobClient: Launched map tasks=7
>> 13/07/11 15:31:32 INFO mapred.JobClient: Data-local map tasks=4
>> 13/07/11 15:31:32 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=21992
>> 13/07/11 15:31:32 INFO mapred.JobClient: File Output Format Counters
>> 13/07/11 15:31:32 INFO mapred.JobClient: Bytes Written=1412505
>> 13/07/11 15:31:32 INFO mapred.JobClient: FileSystemCounters
>> 13/07/11 15:31:32 INFO mapred.JobClient: FILE_BYTES_READ=5414195
>> 13/07/11 15:31:32 INFO mapred.JobClient: HDFS_BYTES_READ=6950820
>> 13/07/11 15:31:32 INFO mapred.JobClient: FILE_BYTES_WRITTEN=8744993
>> 13/07/11 15:31:32 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=1412505
>> 13/07/11 15:31:32 INFO mapred.JobClient: File Input Format Counters
>> 13/07/11 15:31:32 INFO mapred.JobClient: Bytes Read=6950001
>> 13/07/11 15:31:32 INFO mapred.JobClient: Map-Reduce Framework
>> 13/07/11 15:31:32 INFO mapred.JobClient: Map output materialized
>> bytes=3157469
>> 13/07/11 15:31:32 INFO mapred.JobClient: Map input records=137146
>> 13/07/11 15:31:32 INFO mapred.JobClient: Reduce shuffle bytes=2904836
>> 13/07/11 15:31:32 INFO mapred.JobClient: Spilled Records=594764
>> 13/07/11 15:31:32 INFO mapred.JobClient: Map output bytes=11435849
>> 13/07/11 15:31:32 INFO mapred.JobClient: Total committed heap usage
>> (bytes)=1128136704
>> 13/07/11 15:31:32 INFO mapred.JobClient: CPU time spent (ms)=18230
>> 13/07/11 15:31:32 INFO mapred.JobClient: Combine input records=1174991
>> 13/07/11 15:31:32 INFO mapred.JobClient: SPLIT_RAW_BYTES=819
>> 13/07/11 15:31:32 INFO mapred.JobClient: Reduce input records=218990
>> 13/07/11 15:31:32 INFO mapred.JobClient: Reduce input groups=128513
>> 13/07/11 15:31:32 INFO mapred.JobClient: Combine output records=218990
>> 13/07/11 15:31:32 INFO mapred.JobClient: Physical memory (bytes)
>> snapshot=1179656192
>> 13/07/11 15:31:32 INFO mapred.JobClient: Reduce output records=128513
>> 13/07/11 15:31:32 INFO mapred.JobClient: Virtual memory (bytes)
>> snapshot=22992117760
>> 13/07/11 15:31:32 INFO mapred.JobClient: Map output records=1174991
>>
>> from web interface (http://n1:50030/) I saw that both (n1 and n2 ) were
>> used without any errors.
>>
>> Problems appear if I try to use following commands in master (n1):
>>
>> [hduser@n1 ~]$hadoop jar mahout-distribution-0.7/**mahout-examples-0.7-job.jar
>> org.apache.mahout.classifier.**df.mapreduce.BuildForest
>> -Dmapred.max.split.size=**1874231 -p -d testdata/bal_ee_2009.csv -ds
>> testdata/bal_ee_2009.csv.info -sl 10 -o bal_ee_2009_out -t 1
>>
>> SLF4J: Class path contains multiple SLF4J bindings.
>> SLF4J: Found binding in [file:/usr/local/hadoop-1.0.4/**org/slf4j/impl/**
>> StaticLoggerBinder.class]
>> SLF4J: Found binding in [jar:file:/usr/local/hadoop-1.**
>> 0.4/lib/slf4j-log4j12-1.4.3.**jar!/org/slf4j/impl/**
>> StaticLoggerBinder.class]
>> SLF4J: See http://www.slf4j.org/codes.**html#multiple_bindings<http://www.slf4j.org/codes.html#multiple_bindings>for an explanation.
>> 13/07/11 15:36:50 INFO mapreduce.BuildForest: Partial Mapred
>> implementation
>> 13/07/11 15:36:50 INFO mapreduce.BuildForest: Building the forest...
>> 13/07/11 15:36:50 WARN mapred.JobClient: No job jar file set. User
>> classes may not be found. See JobConf(Class) or JobConf#setJar(String).
>> 13/07/11 15:36:50 INFO input.FileInputFormat: Total input paths to
>> process : 1
>> 13/07/11 15:36:50 INFO util.NativeCodeLoader: Loaded the native-hadoop
>> library
>> 13/07/11 15:36:50 WARN snappy.LoadSnappy: Snappy native library not loaded
>> 13/07/11 15:36:50 INFO mapred.JobClient: Running job:
>> job_201307111355_0016
>> 13/07/11 15:36:51 INFO mapred.JobClient: map 0% reduce 0%
>> 13/07/11 15:37:09 INFO mapred.JobClient: map 7% reduce 0%
>> 13/07/11 15:37:11 INFO mapred.JobClient: Task Id :
>> attempt_201307111355_0016_m_**000002_0, Status : FAILED
>> java.lang.RuntimeException: java.lang.**ClassNotFoundException:
>> org.apache.mahout.classifier.**df.mapreduce.partial.**Step1Mapper
>> at org.apache.hadoop.conf.**Configuration.getClass(**
>> Configuration.java:867)
>> at org.apache.hadoop.mapreduce.**JobContext.getMapperClass(**
>> JobContext.java:199)
>> at org.apache.hadoop.mapred.**MapTask.runNewMapper(MapTask.**
>> java:719)
>> at org.apache.hadoop.mapred.**MapTask.run(MapTask.java:370)
>> at org.apache.hadoop.mapred.**Child$4.run(Child.java:255)
>> at java.security.**AccessController.doPrivileged(**Native Method)
>> at javax.security.auth.Subject.**doAs(Subject.java:415)
>> at org.apache.hadoop.security.**UserGroupInformation.doAs(**
>> UserGroupInformation.java:**1121)
>> at org.apache.hadoop.mapred.**Child.main(Child.java:249)
>> Caused by: java.lang.**ClassNotFoundException:
>> org.apache.mahout.classifier.**df.mapreduce.partial.**Step1Mapper
>> at java.net.URLClassLoader$1.run(**URLClassLoader.java:366)
>> at java.net.URLClassLoader$1.run(**URLClassLoader.java:355)
>> at java.security.**AccessController.doPrivileged(**Native Method)
>> at java.net.URLClassLoader.**findClass(URLClassLoader.java:**354)
>> at java.lang.ClassLoader.**loadClass(ClassLoader.java:**424)
>> at sun.misc.Launcher$**AppClassLoader.loadClass(**
>> Launcher.java:308)
>> at java.lang.ClassLoader.**loadClass(ClassLoader.java:**357)
>> at java.lang.Class.forName0(**Native Method)
>> at java.lang.Class.forName(Class.**java:270)
>> at org.apache.hadoop.conf.**Configuration.getClassByName(**
>> Configuration.java:820)
>> at org.apache.hadoop.conf.**Configuration.getClass(**
>> Configuration.java:865)
>> ... 8 more
>>
>> ...
>> ...
>> ...
>>
>> 13/07/11 15:37:37 INFO mapred.JobClient: map 61% reduce 0%
>> 13/07/11 15:37:46 INFO mapred.JobClient: map 76% reduce 0%
>> 13/07/11 15:37:55 INFO mapred.JobClient: map 92% reduce 0%
>> 13/07/11 15:38:01 INFO mapred.JobClient: map 100% reduce 0%
>> 13/07/11 15:38:06 INFO mapred.JobClient: Job complete:
>> job_201307111355_0016
>> 13/07/11 15:38:06 INFO mapred.JobClient: Counters: 20
>> 13/07/11 15:38:06 INFO mapred.JobClient: Job Counters
>> 13/07/11 15:38:06 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=184724
>> 13/07/11 15:38:06 INFO mapred.JobClient: Total time spent by all
>> reduces waiting after reserving slots (ms)=0
>> 13/07/11 15:38:06 INFO mapred.JobClient: Total time spent by all maps
>> waiting after reserving slots (ms)=0
>> 13/07/11 15:38:06 INFO mapred.JobClient: Rack-local map tasks=13
>> 13/07/11 15:38:06 INFO mapred.JobClient: Launched map tasks=22
>> 13/07/11 15:38:06 INFO mapred.JobClient: Data-local map tasks=9
>> 13/07/11 15:38:06 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
>> 13/07/11 15:38:06 INFO mapred.JobClient: File Output Format Counters
>> 13/07/11 15:38:06 INFO mapred.JobClient: Bytes Written=184520
>> 13/07/11 15:38:06 INFO mapred.JobClient: FileSystemCounters
>> 13/07/11 15:38:06 INFO mapred.JobClient: HDFS_BYTES_READ=23444612
>> 13/07/11 15:38:06 INFO mapred.JobClient: FILE_BYTES_WRITTEN=294037
>> 13/07/11 15:38:06 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=184520
>> 13/07/11 15:38:06 INFO mapred.JobClient: File Input Format Counters
>> 13/07/11 15:38:06 INFO mapred.JobClient: Bytes Read=22952913
>> 13/07/11 15:38:06 INFO mapred.JobClient: Map-Reduce Framework
>> 13/07/11 15:38:06 INFO mapred.JobClient: Map input records=204091
>> 13/07/11 15:38:06 INFO mapred.JobClient: Physical memory (bytes)
>> snapshot=1402363904
>> 13/07/11 15:38:06 INFO mapred.JobClient: Spilled Records=0
>> 13/07/11 15:38:06 INFO mapred.JobClient: CPU time spent (ms)=26640
>> 13/07/11 15:38:06 INFO mapred.JobClient: Total committed heap usage
>> (bytes)=1333985280
>> 13/07/11 15:38:06 INFO mapred.JobClient: Virtual memory (bytes)
>> snapshot=37349126144
>> 13/07/11 15:38:06 INFO mapred.JobClient: Map output records=1
>> 13/07/11 15:38:06 INFO mapred.JobClient: SPLIT_RAW_BYTES=1573
>> 13/07/11 15:38:07 INFO common.HadoopUtil: Deleting
>> hdfs://master:54310/user/**hduser/bal_ee_2009_out
>> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Build Time: 0h 1m 16s 977
>> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Forest num Nodes: 10700
>> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Forest mean num Nodes: 10700
>> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Forest mean max Depth: 17
>> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Storing the forest in:
>> bal_ee_2009_out/forest.seq
>>
>> From web console I can see that n1 tasks are fine but n2 tasks failed:
>>
>> java.lang.RuntimeException: java.lang.**ClassNotFoundException:
>> org.apache.mahout.classifier.**df.mapreduce.partial.**Step1Mapper
>> at org.apache.hadoop.conf.**Configuration.getClass(**
>> Configuration.java:867)
>> at org.apache.hadoop.mapreduce.**JobContext.getMapperClass(**
>> JobContext.java:199)
>> at org.apache.hadoop.mapred.**MapTask.runNewMapper(MapTask.**
>> java:719)
>> at org.apache.hadoop.mapred.**MapTask.run(MapTask.java:370)
>> at org.apache.hadoop.mapred.**Child$4.run(Child.java:255)
>> at java.security.**AccessController.doPrivileged(**Native Method)
>> at javax.security.auth.Subject.**doAs(Subject.java:415)
>> at org.apache.hadoop.security.**UserGroupInformation.doAs(**
>> UserGroupInformation.java:**1121)
>> at org.apache.hadoop.mapred.**Child.main(Child.java:249)
>> Caused by: java.lang.**ClassNotFoundException:
>> org.apache.mahout.classifier.**df.mapreduce.partial.**Step1Mapper
>> at java.net.URLClassLoader$1.run(**URLClassLoader.java:366)
>> at java.net.URLClassLoader$1.run(**URLClassLoader.java:355)
>> at java.security.**AccessController.doPrivileged(**Native Method)
>> at java.net.URLClassLoader.**findClass(URLClassLoader.java:**354)
>> at java.lang.ClassLoader.**loadClass(ClassLoader.java:**424)
>> at sun.misc.Launcher$**AppClassLoader.loadClass(**
>> Launcher.java:308)
>> at java.lang.ClassLoader.**loadClass(ClassLoader.java:**357)
>> at java.lang.Class.forName0(**Native Method)
>> at java.lang.Class.forName(Class.**java:270)
>> at org.apache.hadoop.conf.**Configuration.getClassByName(**
>> Configuration.java:820)
>> at org.apache.hadoop.conf.**Configuration.getClass(**
>> Configuration.java:865)
>> ... 8 more
>>
>>
>> But If I run same command from n2
>> [hduser@n2 ~]$ hadoop jar mahout-distribution-0.7/**mahout-examples-0.7-job.jar
>> org.apache.mahout.classifier.**df.mapreduce.BuildForest
>> -Dmapred.max.split.size=**1874231 -p -d testdata/bal_ee_2009.csv -ds
>> testdata/bal_ee_2009.csv.info -sl 10 -o bal_ee_2009_out -t 1
>>
>> 13/07/11 15:41:20 INFO mapreduce.BuildForest: Partial Mapred
>> implementation
>> 13/07/11 15:41:20 INFO mapreduce.BuildForest: Building the forest...
>> 13/07/11 15:41:21 INFO input.FileInputFormat: Total input paths to
>> process : 1
>> 13/07/11 15:41:21 INFO util.NativeCodeLoader: Loaded the native-hadoop
>> library
>> 13/07/11 15:41:21 WARN snappy.LoadSnappy: Snappy native library not loaded
>> 13/07/11 15:41:21 INFO mapred.JobClient: Running job:
>> job_201307111355_0017
>> 13/07/11 15:41:22 INFO mapred.JobClient: map 0% reduce 0%
>> 13/07/11 15:41:43 INFO mapred.JobClient: map 7% reduce 0%
>> 13/07/11 15:41:46 INFO mapred.JobClient: map 15% reduce 0%
>> 13/07/11 15:41:49 INFO mapred.JobClient: map 38% reduce 0%
>> 13/07/11 15:41:52 INFO mapred.JobClient: map 46% reduce 0%
>> 13/07/11 15:42:01 INFO mapred.JobClient: map 61% reduce 0%
>> 13/07/11 15:42:02 INFO mapred.JobClient: map 76% reduce 0%
>> 13/07/11 15:42:09 INFO mapred.JobClient: map 84% reduce 0%
>> 13/07/11 15:42:13 INFO mapred.JobClient: map 100% reduce 0%
>> 13/07/11 15:42:19 INFO mapred.JobClient: Job complete:
>> job_201307111355_0017
>> 13/07/11 15:42:19 INFO mapred.JobClient: Counters: 20
>> 13/07/11 15:42:19 INFO mapred.JobClient: Job Counters
>> 13/07/11 15:42:19 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=138911
>> 13/07/11 15:42:19 INFO mapred.JobClient: Total time spent by all
>> reduces waiting after reserving slots (ms)=0
>> 13/07/11 15:42:19 INFO mapred.JobClient: Total time spent by all maps
>> waiting after reserving slots (ms)=0
>> 13/07/11 15:42:19 INFO mapred.JobClient: Rack-local map tasks=6
>> 13/07/11 15:42:19 INFO mapred.JobClient: Launched map tasks=13
>> 13/07/11 15:42:19 INFO mapred.JobClient: Data-local map tasks=7
>> 13/07/11 15:42:19 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
>> 13/07/11 15:42:19 INFO mapred.JobClient: File Output Format Counters
>> 13/07/11 15:42:19 INFO mapred.JobClient: Bytes Written=184956
>> 13/07/11 15:42:19 INFO mapred.JobClient: FileSystemCounters
>> 13/07/11 15:42:19 INFO mapred.JobClient: HDFS_BYTES_READ=23444612
>> 13/07/11 15:42:19 INFO mapred.JobClient: FILE_BYTES_WRITTEN=296026
>> 13/07/11 15:42:19 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=184956
>> 13/07/11 15:42:19 INFO mapred.JobClient: File Input Format Counters
>> 13/07/11 15:42:19 INFO mapred.JobClient: Bytes Read=22952913
>> 13/07/11 15:42:19 INFO mapred.JobClient: Map-Reduce Framework
>> 13/07/11 15:42:19 INFO mapred.JobClient: Map input records=204091
>> 13/07/11 15:42:19 INFO mapred.JobClient: Physical memory (bytes)
>> snapshot=1370255360
>> 13/07/11 15:42:19 INFO mapred.JobClient: Spilled Records=0
>> 13/07/11 15:42:19 INFO mapred.JobClient: CPU time spent (ms)=27990
>> 13/07/11 15:42:19 INFO mapred.JobClient: Total committed heap usage
>> (bytes)=1292500992
>> 13/07/11 15:42:19 INFO mapred.JobClient: Virtual memory (bytes)
>> snapshot=37360082944
>> 13/07/11 15:42:19 INFO mapred.JobClient: Map output records=1
>> 13/07/11 15:42:19 INFO mapred.JobClient: SPLIT_RAW_BYTES=1573
>> 13/07/11 15:42:20 INFO common.HadoopUtil: Deleting
>> hdfs://master:54310/user/**hduser/bal_ee_2009_out
>> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Build Time: 0h 0m 59s 386
>> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Forest num Nodes: 10698
>> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Forest mean num Nodes: 10698
>> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Forest mean max Depth: 16
>> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Storing the forest in:
>> bal_ee_2009_out/forest.seq
>>
>> Both (n1 and n2) are used and from web console I can see that there are
>> no errors.
>>
>> Is there any explanations why I am getting errors when I run command from
>> master?
>>
>>
>> --
>> Regards, Margus (Margusja) Roo
>> +372 51 48 780
>> http://margus.roo.ee
>> skype: margusja
>> -----BEGIN PUBLIC KEY-----
>> MIGfMA0GCSqGSIb3DQEBAQUAA4GNAD**CBiQKBgQCvbeg7LwEC2SCpAEewwpC3**ajxE
>> 5ZsRMCB77L8bae9G7TslgLkoIzo9yO**jPdx2NN6DllKbV65UjTay43uUDyql9**g3tl
>> RhiJIcoAExkSTykWqAIPR88LfilLy1**JlQ+0RD8OXiWOVVQfhOHpQ0R/**jcAkM2lZa
>> BjM8j36yJvoBVsfOHQIDAQAB
>> -----END PUBLIC KEY-----
>>
>>
Re: Task failure in slave node
Posted by Azuryy Yu <az...@gmail.com>.
sorry for typo,
mahout, not mahou. sent from mobile
On Jul 11, 2013 9:40 PM, "Azuryy Yu" <az...@gmail.com> wrote:
> hi,
>
> put all mahou jars under hadoop_home/lib, then restart cluster.
> On Jul 11, 2013 8:45 PM, "Margusja" <ma...@roo.ee> wrote:
>
>> Hi
>>
>> I have tow nodes:
>> n1 (master, salve) and n2 (slave)
>>
>> after set up I ran wordcount example and it worked fine:
>> [hduser@n1 ~]$ hadoop jar /usr/local/hadoop/hadoop-**examples-1.0.4.jar
>> wordcount /user/hduser/gutenberg /user/hduser/gutenberg-output
>> 13/07/11 15:30:44 INFO input.FileInputFormat: Total input paths to
>> process : 7
>> 13/07/11 15:30:44 INFO util.NativeCodeLoader: Loaded the native-hadoop
>> library
>> 13/07/11 15:30:44 WARN snappy.LoadSnappy: Snappy native library not loaded
>> 13/07/11 15:30:44 INFO mapred.JobClient: Running job:
>> job_201307111355_0015
>> 13/07/11 15:30:45 INFO mapred.JobClient: map 0% reduce 0%
>> 13/07/11 15:31:03 INFO mapred.JobClient: map 42% reduce 0%
>> 13/07/11 15:31:06 INFO mapred.JobClient: map 57% reduce 0%
>> 13/07/11 15:31:09 INFO mapred.JobClient: map 71% reduce 0%
>> 13/07/11 15:31:15 INFO mapred.JobClient: map 100% reduce 0%
>> 13/07/11 15:31:18 INFO mapred.JobClient: map 100% reduce 23%
>> 13/07/11 15:31:27 INFO mapred.JobClient: map 100% reduce 100%
>> 13/07/11 15:31:32 INFO mapred.JobClient: Job complete:
>> job_201307111355_0015
>> 13/07/11 15:31:32 INFO mapred.JobClient: Counters: 30
>> 13/07/11 15:31:32 INFO mapred.JobClient: Job Counters
>> 13/07/11 15:31:32 INFO mapred.JobClient: Launched reduce tasks=1
>> 13/07/11 15:31:32 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=67576
>> 13/07/11 15:31:32 INFO mapred.JobClient: Total time spent by all
>> reduces waiting after reserving slots (ms)=0
>> 13/07/11 15:31:32 INFO mapred.JobClient: Total time spent by all maps
>> waiting after reserving slots (ms)=0
>> 13/07/11 15:31:32 INFO mapred.JobClient: Rack-local map tasks=3
>> 13/07/11 15:31:32 INFO mapred.JobClient: Launched map tasks=7
>> 13/07/11 15:31:32 INFO mapred.JobClient: Data-local map tasks=4
>> 13/07/11 15:31:32 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=21992
>> 13/07/11 15:31:32 INFO mapred.JobClient: File Output Format Counters
>> 13/07/11 15:31:32 INFO mapred.JobClient: Bytes Written=1412505
>> 13/07/11 15:31:32 INFO mapred.JobClient: FileSystemCounters
>> 13/07/11 15:31:32 INFO mapred.JobClient: FILE_BYTES_READ=5414195
>> 13/07/11 15:31:32 INFO mapred.JobClient: HDFS_BYTES_READ=6950820
>> 13/07/11 15:31:32 INFO mapred.JobClient: FILE_BYTES_WRITTEN=8744993
>> 13/07/11 15:31:32 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=1412505
>> 13/07/11 15:31:32 INFO mapred.JobClient: File Input Format Counters
>> 13/07/11 15:31:32 INFO mapred.JobClient: Bytes Read=6950001
>> 13/07/11 15:31:32 INFO mapred.JobClient: Map-Reduce Framework
>> 13/07/11 15:31:32 INFO mapred.JobClient: Map output materialized
>> bytes=3157469
>> 13/07/11 15:31:32 INFO mapred.JobClient: Map input records=137146
>> 13/07/11 15:31:32 INFO mapred.JobClient: Reduce shuffle bytes=2904836
>> 13/07/11 15:31:32 INFO mapred.JobClient: Spilled Records=594764
>> 13/07/11 15:31:32 INFO mapred.JobClient: Map output bytes=11435849
>> 13/07/11 15:31:32 INFO mapred.JobClient: Total committed heap usage
>> (bytes)=1128136704
>> 13/07/11 15:31:32 INFO mapred.JobClient: CPU time spent (ms)=18230
>> 13/07/11 15:31:32 INFO mapred.JobClient: Combine input records=1174991
>> 13/07/11 15:31:32 INFO mapred.JobClient: SPLIT_RAW_BYTES=819
>> 13/07/11 15:31:32 INFO mapred.JobClient: Reduce input records=218990
>> 13/07/11 15:31:32 INFO mapred.JobClient: Reduce input groups=128513
>> 13/07/11 15:31:32 INFO mapred.JobClient: Combine output records=218990
>> 13/07/11 15:31:32 INFO mapred.JobClient: Physical memory (bytes)
>> snapshot=1179656192
>> 13/07/11 15:31:32 INFO mapred.JobClient: Reduce output records=128513
>> 13/07/11 15:31:32 INFO mapred.JobClient: Virtual memory (bytes)
>> snapshot=22992117760
>> 13/07/11 15:31:32 INFO mapred.JobClient: Map output records=1174991
>>
>> from web interface (http://n1:50030/) I saw that both (n1 and n2 ) were
>> used without any errors.
>>
>> Problems appear if I try to use following commands in master (n1):
>>
>> [hduser@n1 ~]$hadoop jar mahout-distribution-0.7/**mahout-examples-0.7-job.jar
>> org.apache.mahout.classifier.**df.mapreduce.BuildForest
>> -Dmapred.max.split.size=**1874231 -p -d testdata/bal_ee_2009.csv -ds
>> testdata/bal_ee_2009.csv.info -sl 10 -o bal_ee_2009_out -t 1
>>
>> SLF4J: Class path contains multiple SLF4J bindings.
>> SLF4J: Found binding in [file:/usr/local/hadoop-1.0.4/**org/slf4j/impl/**
>> StaticLoggerBinder.class]
>> SLF4J: Found binding in [jar:file:/usr/local/hadoop-1.**
>> 0.4/lib/slf4j-log4j12-1.4.3.**jar!/org/slf4j/impl/**
>> StaticLoggerBinder.class]
>> SLF4J: See http://www.slf4j.org/codes.**html#multiple_bindings<http://www.slf4j.org/codes.html#multiple_bindings>for an explanation.
>> 13/07/11 15:36:50 INFO mapreduce.BuildForest: Partial Mapred
>> implementation
>> 13/07/11 15:36:50 INFO mapreduce.BuildForest: Building the forest...
>> 13/07/11 15:36:50 WARN mapred.JobClient: No job jar file set. User
>> classes may not be found. See JobConf(Class) or JobConf#setJar(String).
>> 13/07/11 15:36:50 INFO input.FileInputFormat: Total input paths to
>> process : 1
>> 13/07/11 15:36:50 INFO util.NativeCodeLoader: Loaded the native-hadoop
>> library
>> 13/07/11 15:36:50 WARN snappy.LoadSnappy: Snappy native library not loaded
>> 13/07/11 15:36:50 INFO mapred.JobClient: Running job:
>> job_201307111355_0016
>> 13/07/11 15:36:51 INFO mapred.JobClient: map 0% reduce 0%
>> 13/07/11 15:37:09 INFO mapred.JobClient: map 7% reduce 0%
>> 13/07/11 15:37:11 INFO mapred.JobClient: Task Id :
>> attempt_201307111355_0016_m_**000002_0, Status : FAILED
>> java.lang.RuntimeException: java.lang.**ClassNotFoundException:
>> org.apache.mahout.classifier.**df.mapreduce.partial.**Step1Mapper
>> at org.apache.hadoop.conf.**Configuration.getClass(**
>> Configuration.java:867)
>> at org.apache.hadoop.mapreduce.**JobContext.getMapperClass(**
>> JobContext.java:199)
>> at org.apache.hadoop.mapred.**MapTask.runNewMapper(MapTask.**
>> java:719)
>> at org.apache.hadoop.mapred.**MapTask.run(MapTask.java:370)
>> at org.apache.hadoop.mapred.**Child$4.run(Child.java:255)
>> at java.security.**AccessController.doPrivileged(**Native Method)
>> at javax.security.auth.Subject.**doAs(Subject.java:415)
>> at org.apache.hadoop.security.**UserGroupInformation.doAs(**
>> UserGroupInformation.java:**1121)
>> at org.apache.hadoop.mapred.**Child.main(Child.java:249)
>> Caused by: java.lang.**ClassNotFoundException:
>> org.apache.mahout.classifier.**df.mapreduce.partial.**Step1Mapper
>> at java.net.URLClassLoader$1.run(**URLClassLoader.java:366)
>> at java.net.URLClassLoader$1.run(**URLClassLoader.java:355)
>> at java.security.**AccessController.doPrivileged(**Native Method)
>> at java.net.URLClassLoader.**findClass(URLClassLoader.java:**354)
>> at java.lang.ClassLoader.**loadClass(ClassLoader.java:**424)
>> at sun.misc.Launcher$**AppClassLoader.loadClass(**
>> Launcher.java:308)
>> at java.lang.ClassLoader.**loadClass(ClassLoader.java:**357)
>> at java.lang.Class.forName0(**Native Method)
>> at java.lang.Class.forName(Class.**java:270)
>> at org.apache.hadoop.conf.**Configuration.getClassByName(**
>> Configuration.java:820)
>> at org.apache.hadoop.conf.**Configuration.getClass(**
>> Configuration.java:865)
>> ... 8 more
>>
>> ...
>> ...
>> ...
>>
>> 13/07/11 15:37:37 INFO mapred.JobClient: map 61% reduce 0%
>> 13/07/11 15:37:46 INFO mapred.JobClient: map 76% reduce 0%
>> 13/07/11 15:37:55 INFO mapred.JobClient: map 92% reduce 0%
>> 13/07/11 15:38:01 INFO mapred.JobClient: map 100% reduce 0%
>> 13/07/11 15:38:06 INFO mapred.JobClient: Job complete:
>> job_201307111355_0016
>> 13/07/11 15:38:06 INFO mapred.JobClient: Counters: 20
>> 13/07/11 15:38:06 INFO mapred.JobClient: Job Counters
>> 13/07/11 15:38:06 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=184724
>> 13/07/11 15:38:06 INFO mapred.JobClient: Total time spent by all
>> reduces waiting after reserving slots (ms)=0
>> 13/07/11 15:38:06 INFO mapred.JobClient: Total time spent by all maps
>> waiting after reserving slots (ms)=0
>> 13/07/11 15:38:06 INFO mapred.JobClient: Rack-local map tasks=13
>> 13/07/11 15:38:06 INFO mapred.JobClient: Launched map tasks=22
>> 13/07/11 15:38:06 INFO mapred.JobClient: Data-local map tasks=9
>> 13/07/11 15:38:06 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
>> 13/07/11 15:38:06 INFO mapred.JobClient: File Output Format Counters
>> 13/07/11 15:38:06 INFO mapred.JobClient: Bytes Written=184520
>> 13/07/11 15:38:06 INFO mapred.JobClient: FileSystemCounters
>> 13/07/11 15:38:06 INFO mapred.JobClient: HDFS_BYTES_READ=23444612
>> 13/07/11 15:38:06 INFO mapred.JobClient: FILE_BYTES_WRITTEN=294037
>> 13/07/11 15:38:06 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=184520
>> 13/07/11 15:38:06 INFO mapred.JobClient: File Input Format Counters
>> 13/07/11 15:38:06 INFO mapred.JobClient: Bytes Read=22952913
>> 13/07/11 15:38:06 INFO mapred.JobClient: Map-Reduce Framework
>> 13/07/11 15:38:06 INFO mapred.JobClient: Map input records=204091
>> 13/07/11 15:38:06 INFO mapred.JobClient: Physical memory (bytes)
>> snapshot=1402363904
>> 13/07/11 15:38:06 INFO mapred.JobClient: Spilled Records=0
>> 13/07/11 15:38:06 INFO mapred.JobClient: CPU time spent (ms)=26640
>> 13/07/11 15:38:06 INFO mapred.JobClient: Total committed heap usage
>> (bytes)=1333985280
>> 13/07/11 15:38:06 INFO mapred.JobClient: Virtual memory (bytes)
>> snapshot=37349126144
>> 13/07/11 15:38:06 INFO mapred.JobClient: Map output records=1
>> 13/07/11 15:38:06 INFO mapred.JobClient: SPLIT_RAW_BYTES=1573
>> 13/07/11 15:38:07 INFO common.HadoopUtil: Deleting
>> hdfs://master:54310/user/**hduser/bal_ee_2009_out
>> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Build Time: 0h 1m 16s 977
>> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Forest num Nodes: 10700
>> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Forest mean num Nodes: 10700
>> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Forest mean max Depth: 17
>> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Storing the forest in:
>> bal_ee_2009_out/forest.seq
>>
>> From web console I can see that n1 tasks are fine but n2 tasks failed:
>>
>> java.lang.RuntimeException: java.lang.**ClassNotFoundException:
>> org.apache.mahout.classifier.**df.mapreduce.partial.**Step1Mapper
>> at org.apache.hadoop.conf.**Configuration.getClass(**
>> Configuration.java:867)
>> at org.apache.hadoop.mapreduce.**JobContext.getMapperClass(**
>> JobContext.java:199)
>> at org.apache.hadoop.mapred.**MapTask.runNewMapper(MapTask.**
>> java:719)
>> at org.apache.hadoop.mapred.**MapTask.run(MapTask.java:370)
>> at org.apache.hadoop.mapred.**Child$4.run(Child.java:255)
>> at java.security.**AccessController.doPrivileged(**Native Method)
>> at javax.security.auth.Subject.**doAs(Subject.java:415)
>> at org.apache.hadoop.security.**UserGroupInformation.doAs(**
>> UserGroupInformation.java:**1121)
>> at org.apache.hadoop.mapred.**Child.main(Child.java:249)
>> Caused by: java.lang.**ClassNotFoundException:
>> org.apache.mahout.classifier.**df.mapreduce.partial.**Step1Mapper
>> at java.net.URLClassLoader$1.run(**URLClassLoader.java:366)
>> at java.net.URLClassLoader$1.run(**URLClassLoader.java:355)
>> at java.security.**AccessController.doPrivileged(**Native Method)
>> at java.net.URLClassLoader.**findClass(URLClassLoader.java:**354)
>> at java.lang.ClassLoader.**loadClass(ClassLoader.java:**424)
>> at sun.misc.Launcher$**AppClassLoader.loadClass(**
>> Launcher.java:308)
>> at java.lang.ClassLoader.**loadClass(ClassLoader.java:**357)
>> at java.lang.Class.forName0(**Native Method)
>> at java.lang.Class.forName(Class.**java:270)
>> at org.apache.hadoop.conf.**Configuration.getClassByName(**
>> Configuration.java:820)
>> at org.apache.hadoop.conf.**Configuration.getClass(**
>> Configuration.java:865)
>> ... 8 more
>>
>>
>> But If I run same command from n2
>> [hduser@n2 ~]$ hadoop jar mahout-distribution-0.7/**mahout-examples-0.7-job.jar
>> org.apache.mahout.classifier.**df.mapreduce.BuildForest
>> -Dmapred.max.split.size=**1874231 -p -d testdata/bal_ee_2009.csv -ds
>> testdata/bal_ee_2009.csv.info -sl 10 -o bal_ee_2009_out -t 1
>>
>> 13/07/11 15:41:20 INFO mapreduce.BuildForest: Partial Mapred
>> implementation
>> 13/07/11 15:41:20 INFO mapreduce.BuildForest: Building the forest...
>> 13/07/11 15:41:21 INFO input.FileInputFormat: Total input paths to
>> process : 1
>> 13/07/11 15:41:21 INFO util.NativeCodeLoader: Loaded the native-hadoop
>> library
>> 13/07/11 15:41:21 WARN snappy.LoadSnappy: Snappy native library not loaded
>> 13/07/11 15:41:21 INFO mapred.JobClient: Running job:
>> job_201307111355_0017
>> 13/07/11 15:41:22 INFO mapred.JobClient: map 0% reduce 0%
>> 13/07/11 15:41:43 INFO mapred.JobClient: map 7% reduce 0%
>> 13/07/11 15:41:46 INFO mapred.JobClient: map 15% reduce 0%
>> 13/07/11 15:41:49 INFO mapred.JobClient: map 38% reduce 0%
>> 13/07/11 15:41:52 INFO mapred.JobClient: map 46% reduce 0%
>> 13/07/11 15:42:01 INFO mapred.JobClient: map 61% reduce 0%
>> 13/07/11 15:42:02 INFO mapred.JobClient: map 76% reduce 0%
>> 13/07/11 15:42:09 INFO mapred.JobClient: map 84% reduce 0%
>> 13/07/11 15:42:13 INFO mapred.JobClient: map 100% reduce 0%
>> 13/07/11 15:42:19 INFO mapred.JobClient: Job complete:
>> job_201307111355_0017
>> 13/07/11 15:42:19 INFO mapred.JobClient: Counters: 20
>> 13/07/11 15:42:19 INFO mapred.JobClient: Job Counters
>> 13/07/11 15:42:19 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=138911
>> 13/07/11 15:42:19 INFO mapred.JobClient: Total time spent by all
>> reduces waiting after reserving slots (ms)=0
>> 13/07/11 15:42:19 INFO mapred.JobClient: Total time spent by all maps
>> waiting after reserving slots (ms)=0
>> 13/07/11 15:42:19 INFO mapred.JobClient: Rack-local map tasks=6
>> 13/07/11 15:42:19 INFO mapred.JobClient: Launched map tasks=13
>> 13/07/11 15:42:19 INFO mapred.JobClient: Data-local map tasks=7
>> 13/07/11 15:42:19 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
>> 13/07/11 15:42:19 INFO mapred.JobClient: File Output Format Counters
>> 13/07/11 15:42:19 INFO mapred.JobClient: Bytes Written=184956
>> 13/07/11 15:42:19 INFO mapred.JobClient: FileSystemCounters
>> 13/07/11 15:42:19 INFO mapred.JobClient: HDFS_BYTES_READ=23444612
>> 13/07/11 15:42:19 INFO mapred.JobClient: FILE_BYTES_WRITTEN=296026
>> 13/07/11 15:42:19 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=184956
>> 13/07/11 15:42:19 INFO mapred.JobClient: File Input Format Counters
>> 13/07/11 15:42:19 INFO mapred.JobClient: Bytes Read=22952913
>> 13/07/11 15:42:19 INFO mapred.JobClient: Map-Reduce Framework
>> 13/07/11 15:42:19 INFO mapred.JobClient: Map input records=204091
>> 13/07/11 15:42:19 INFO mapred.JobClient: Physical memory (bytes)
>> snapshot=1370255360
>> 13/07/11 15:42:19 INFO mapred.JobClient: Spilled Records=0
>> 13/07/11 15:42:19 INFO mapred.JobClient: CPU time spent (ms)=27990
>> 13/07/11 15:42:19 INFO mapred.JobClient: Total committed heap usage
>> (bytes)=1292500992
>> 13/07/11 15:42:19 INFO mapred.JobClient: Virtual memory (bytes)
>> snapshot=37360082944
>> 13/07/11 15:42:19 INFO mapred.JobClient: Map output records=1
>> 13/07/11 15:42:19 INFO mapred.JobClient: SPLIT_RAW_BYTES=1573
>> 13/07/11 15:42:20 INFO common.HadoopUtil: Deleting
>> hdfs://master:54310/user/**hduser/bal_ee_2009_out
>> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Build Time: 0h 0m 59s 386
>> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Forest num Nodes: 10698
>> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Forest mean num Nodes: 10698
>> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Forest mean max Depth: 16
>> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Storing the forest in:
>> bal_ee_2009_out/forest.seq
>>
>> Both (n1 and n2) are used and from web console I can see that there are
>> no errors.
>>
>> Is there any explanations why I am getting errors when I run command from
>> master?
>>
>>
>> --
>> Regards, Margus (Margusja) Roo
>> +372 51 48 780
>> http://margus.roo.ee
>> skype: margusja
>> -----BEGIN PUBLIC KEY-----
>> MIGfMA0GCSqGSIb3DQEBAQUAA4GNAD**CBiQKBgQCvbeg7LwEC2SCpAEewwpC3**ajxE
>> 5ZsRMCB77L8bae9G7TslgLkoIzo9yO**jPdx2NN6DllKbV65UjTay43uUDyql9**g3tl
>> RhiJIcoAExkSTykWqAIPR88LfilLy1**JlQ+0RD8OXiWOVVQfhOHpQ0R/**jcAkM2lZa
>> BjM8j36yJvoBVsfOHQIDAQAB
>> -----END PUBLIC KEY-----
>>
>>
Re: Task failure in slave node
Posted by Azuryy Yu <az...@gmail.com>.
sorry for typo,
mahout, not mahou. sent from mobile
On Jul 11, 2013 9:40 PM, "Azuryy Yu" <az...@gmail.com> wrote:
> hi,
>
> put all mahou jars under hadoop_home/lib, then restart cluster.
> On Jul 11, 2013 8:45 PM, "Margusja" <ma...@roo.ee> wrote:
>
>> Hi
>>
>> I have tow nodes:
>> n1 (master, salve) and n2 (slave)
>>
>> after set up I ran wordcount example and it worked fine:
>> [hduser@n1 ~]$ hadoop jar /usr/local/hadoop/hadoop-**examples-1.0.4.jar
>> wordcount /user/hduser/gutenberg /user/hduser/gutenberg-output
>> 13/07/11 15:30:44 INFO input.FileInputFormat: Total input paths to
>> process : 7
>> 13/07/11 15:30:44 INFO util.NativeCodeLoader: Loaded the native-hadoop
>> library
>> 13/07/11 15:30:44 WARN snappy.LoadSnappy: Snappy native library not loaded
>> 13/07/11 15:30:44 INFO mapred.JobClient: Running job:
>> job_201307111355_0015
>> 13/07/11 15:30:45 INFO mapred.JobClient: map 0% reduce 0%
>> 13/07/11 15:31:03 INFO mapred.JobClient: map 42% reduce 0%
>> 13/07/11 15:31:06 INFO mapred.JobClient: map 57% reduce 0%
>> 13/07/11 15:31:09 INFO mapred.JobClient: map 71% reduce 0%
>> 13/07/11 15:31:15 INFO mapred.JobClient: map 100% reduce 0%
>> 13/07/11 15:31:18 INFO mapred.JobClient: map 100% reduce 23%
>> 13/07/11 15:31:27 INFO mapred.JobClient: map 100% reduce 100%
>> 13/07/11 15:31:32 INFO mapred.JobClient: Job complete:
>> job_201307111355_0015
>> 13/07/11 15:31:32 INFO mapred.JobClient: Counters: 30
>> 13/07/11 15:31:32 INFO mapred.JobClient: Job Counters
>> 13/07/11 15:31:32 INFO mapred.JobClient: Launched reduce tasks=1
>> 13/07/11 15:31:32 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=67576
>> 13/07/11 15:31:32 INFO mapred.JobClient: Total time spent by all
>> reduces waiting after reserving slots (ms)=0
>> 13/07/11 15:31:32 INFO mapred.JobClient: Total time spent by all maps
>> waiting after reserving slots (ms)=0
>> 13/07/11 15:31:32 INFO mapred.JobClient: Rack-local map tasks=3
>> 13/07/11 15:31:32 INFO mapred.JobClient: Launched map tasks=7
>> 13/07/11 15:31:32 INFO mapred.JobClient: Data-local map tasks=4
>> 13/07/11 15:31:32 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=21992
>> 13/07/11 15:31:32 INFO mapred.JobClient: File Output Format Counters
>> 13/07/11 15:31:32 INFO mapred.JobClient: Bytes Written=1412505
>> 13/07/11 15:31:32 INFO mapred.JobClient: FileSystemCounters
>> 13/07/11 15:31:32 INFO mapred.JobClient: FILE_BYTES_READ=5414195
>> 13/07/11 15:31:32 INFO mapred.JobClient: HDFS_BYTES_READ=6950820
>> 13/07/11 15:31:32 INFO mapred.JobClient: FILE_BYTES_WRITTEN=8744993
>> 13/07/11 15:31:32 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=1412505
>> 13/07/11 15:31:32 INFO mapred.JobClient: File Input Format Counters
>> 13/07/11 15:31:32 INFO mapred.JobClient: Bytes Read=6950001
>> 13/07/11 15:31:32 INFO mapred.JobClient: Map-Reduce Framework
>> 13/07/11 15:31:32 INFO mapred.JobClient: Map output materialized
>> bytes=3157469
>> 13/07/11 15:31:32 INFO mapred.JobClient: Map input records=137146
>> 13/07/11 15:31:32 INFO mapred.JobClient: Reduce shuffle bytes=2904836
>> 13/07/11 15:31:32 INFO mapred.JobClient: Spilled Records=594764
>> 13/07/11 15:31:32 INFO mapred.JobClient: Map output bytes=11435849
>> 13/07/11 15:31:32 INFO mapred.JobClient: Total committed heap usage
>> (bytes)=1128136704
>> 13/07/11 15:31:32 INFO mapred.JobClient: CPU time spent (ms)=18230
>> 13/07/11 15:31:32 INFO mapred.JobClient: Combine input records=1174991
>> 13/07/11 15:31:32 INFO mapred.JobClient: SPLIT_RAW_BYTES=819
>> 13/07/11 15:31:32 INFO mapred.JobClient: Reduce input records=218990
>> 13/07/11 15:31:32 INFO mapred.JobClient: Reduce input groups=128513
>> 13/07/11 15:31:32 INFO mapred.JobClient: Combine output records=218990
>> 13/07/11 15:31:32 INFO mapred.JobClient: Physical memory (bytes)
>> snapshot=1179656192
>> 13/07/11 15:31:32 INFO mapred.JobClient: Reduce output records=128513
>> 13/07/11 15:31:32 INFO mapred.JobClient: Virtual memory (bytes)
>> snapshot=22992117760
>> 13/07/11 15:31:32 INFO mapred.JobClient: Map output records=1174991
>>
>> from web interface (http://n1:50030/) I saw that both (n1 and n2 ) were
>> used without any errors.
>>
>> Problems appear if I try to use following commands in master (n1):
>>
>> [hduser@n1 ~]$hadoop jar mahout-distribution-0.7/**mahout-examples-0.7-job.jar
>> org.apache.mahout.classifier.**df.mapreduce.BuildForest
>> -Dmapred.max.split.size=**1874231 -p -d testdata/bal_ee_2009.csv -ds
>> testdata/bal_ee_2009.csv.info -sl 10 -o bal_ee_2009_out -t 1
>>
>> SLF4J: Class path contains multiple SLF4J bindings.
>> SLF4J: Found binding in [file:/usr/local/hadoop-1.0.4/**org/slf4j/impl/**
>> StaticLoggerBinder.class]
>> SLF4J: Found binding in [jar:file:/usr/local/hadoop-1.**
>> 0.4/lib/slf4j-log4j12-1.4.3.**jar!/org/slf4j/impl/**
>> StaticLoggerBinder.class]
>> SLF4J: See http://www.slf4j.org/codes.**html#multiple_bindings<http://www.slf4j.org/codes.html#multiple_bindings>for an explanation.
>> 13/07/11 15:36:50 INFO mapreduce.BuildForest: Partial Mapred
>> implementation
>> 13/07/11 15:36:50 INFO mapreduce.BuildForest: Building the forest...
>> 13/07/11 15:36:50 WARN mapred.JobClient: No job jar file set. User
>> classes may not be found. See JobConf(Class) or JobConf#setJar(String).
>> 13/07/11 15:36:50 INFO input.FileInputFormat: Total input paths to
>> process : 1
>> 13/07/11 15:36:50 INFO util.NativeCodeLoader: Loaded the native-hadoop
>> library
>> 13/07/11 15:36:50 WARN snappy.LoadSnappy: Snappy native library not loaded
>> 13/07/11 15:36:50 INFO mapred.JobClient: Running job:
>> job_201307111355_0016
>> 13/07/11 15:36:51 INFO mapred.JobClient: map 0% reduce 0%
>> 13/07/11 15:37:09 INFO mapred.JobClient: map 7% reduce 0%
>> 13/07/11 15:37:11 INFO mapred.JobClient: Task Id :
>> attempt_201307111355_0016_m_**000002_0, Status : FAILED
>> java.lang.RuntimeException: java.lang.**ClassNotFoundException:
>> org.apache.mahout.classifier.**df.mapreduce.partial.**Step1Mapper
>> at org.apache.hadoop.conf.**Configuration.getClass(**
>> Configuration.java:867)
>> at org.apache.hadoop.mapreduce.**JobContext.getMapperClass(**
>> JobContext.java:199)
>> at org.apache.hadoop.mapred.**MapTask.runNewMapper(MapTask.**
>> java:719)
>> at org.apache.hadoop.mapred.**MapTask.run(MapTask.java:370)
>> at org.apache.hadoop.mapred.**Child$4.run(Child.java:255)
>> at java.security.**AccessController.doPrivileged(**Native Method)
>> at javax.security.auth.Subject.**doAs(Subject.java:415)
>> at org.apache.hadoop.security.**UserGroupInformation.doAs(**
>> UserGroupInformation.java:**1121)
>> at org.apache.hadoop.mapred.**Child.main(Child.java:249)
>> Caused by: java.lang.**ClassNotFoundException:
>> org.apache.mahout.classifier.**df.mapreduce.partial.**Step1Mapper
>> at java.net.URLClassLoader$1.run(**URLClassLoader.java:366)
>> at java.net.URLClassLoader$1.run(**URLClassLoader.java:355)
>> at java.security.**AccessController.doPrivileged(**Native Method)
>> at java.net.URLClassLoader.**findClass(URLClassLoader.java:**354)
>> at java.lang.ClassLoader.**loadClass(ClassLoader.java:**424)
>> at sun.misc.Launcher$**AppClassLoader.loadClass(**
>> Launcher.java:308)
>> at java.lang.ClassLoader.**loadClass(ClassLoader.java:**357)
>> at java.lang.Class.forName0(**Native Method)
>> at java.lang.Class.forName(Class.**java:270)
>> at org.apache.hadoop.conf.**Configuration.getClassByName(**
>> Configuration.java:820)
>> at org.apache.hadoop.conf.**Configuration.getClass(**
>> Configuration.java:865)
>> ... 8 more
>>
>> ...
>> ...
>> ...
>>
>> 13/07/11 15:37:37 INFO mapred.JobClient: map 61% reduce 0%
>> 13/07/11 15:37:46 INFO mapred.JobClient: map 76% reduce 0%
>> 13/07/11 15:37:55 INFO mapred.JobClient: map 92% reduce 0%
>> 13/07/11 15:38:01 INFO mapred.JobClient: map 100% reduce 0%
>> 13/07/11 15:38:06 INFO mapred.JobClient: Job complete:
>> job_201307111355_0016
>> 13/07/11 15:38:06 INFO mapred.JobClient: Counters: 20
>> 13/07/11 15:38:06 INFO mapred.JobClient: Job Counters
>> 13/07/11 15:38:06 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=184724
>> 13/07/11 15:38:06 INFO mapred.JobClient: Total time spent by all
>> reduces waiting after reserving slots (ms)=0
>> 13/07/11 15:38:06 INFO mapred.JobClient: Total time spent by all maps
>> waiting after reserving slots (ms)=0
>> 13/07/11 15:38:06 INFO mapred.JobClient: Rack-local map tasks=13
>> 13/07/11 15:38:06 INFO mapred.JobClient: Launched map tasks=22
>> 13/07/11 15:38:06 INFO mapred.JobClient: Data-local map tasks=9
>> 13/07/11 15:38:06 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
>> 13/07/11 15:38:06 INFO mapred.JobClient: File Output Format Counters
>> 13/07/11 15:38:06 INFO mapred.JobClient: Bytes Written=184520
>> 13/07/11 15:38:06 INFO mapred.JobClient: FileSystemCounters
>> 13/07/11 15:38:06 INFO mapred.JobClient: HDFS_BYTES_READ=23444612
>> 13/07/11 15:38:06 INFO mapred.JobClient: FILE_BYTES_WRITTEN=294037
>> 13/07/11 15:38:06 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=184520
>> 13/07/11 15:38:06 INFO mapred.JobClient: File Input Format Counters
>> 13/07/11 15:38:06 INFO mapred.JobClient: Bytes Read=22952913
>> 13/07/11 15:38:06 INFO mapred.JobClient: Map-Reduce Framework
>> 13/07/11 15:38:06 INFO mapred.JobClient: Map input records=204091
>> 13/07/11 15:38:06 INFO mapred.JobClient: Physical memory (bytes)
>> snapshot=1402363904
>> 13/07/11 15:38:06 INFO mapred.JobClient: Spilled Records=0
>> 13/07/11 15:38:06 INFO mapred.JobClient: CPU time spent (ms)=26640
>> 13/07/11 15:38:06 INFO mapred.JobClient: Total committed heap usage
>> (bytes)=1333985280
>> 13/07/11 15:38:06 INFO mapred.JobClient: Virtual memory (bytes)
>> snapshot=37349126144
>> 13/07/11 15:38:06 INFO mapred.JobClient: Map output records=1
>> 13/07/11 15:38:06 INFO mapred.JobClient: SPLIT_RAW_BYTES=1573
>> 13/07/11 15:38:07 INFO common.HadoopUtil: Deleting
>> hdfs://master:54310/user/**hduser/bal_ee_2009_out
>> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Build Time: 0h 1m 16s 977
>> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Forest num Nodes: 10700
>> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Forest mean num Nodes: 10700
>> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Forest mean max Depth: 17
>> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Storing the forest in:
>> bal_ee_2009_out/forest.seq
>>
>> From web console I can see that n1 tasks are fine but n2 tasks failed:
>>
>> java.lang.RuntimeException: java.lang.**ClassNotFoundException:
>> org.apache.mahout.classifier.**df.mapreduce.partial.**Step1Mapper
>> at org.apache.hadoop.conf.**Configuration.getClass(**
>> Configuration.java:867)
>> at org.apache.hadoop.mapreduce.**JobContext.getMapperClass(**
>> JobContext.java:199)
>> at org.apache.hadoop.mapred.**MapTask.runNewMapper(MapTask.**
>> java:719)
>> at org.apache.hadoop.mapred.**MapTask.run(MapTask.java:370)
>> at org.apache.hadoop.mapred.**Child$4.run(Child.java:255)
>> at java.security.**AccessController.doPrivileged(**Native Method)
>> at javax.security.auth.Subject.**doAs(Subject.java:415)
>> at org.apache.hadoop.security.**UserGroupInformation.doAs(**
>> UserGroupInformation.java:**1121)
>> at org.apache.hadoop.mapred.**Child.main(Child.java:249)
>> Caused by: java.lang.**ClassNotFoundException:
>> org.apache.mahout.classifier.**df.mapreduce.partial.**Step1Mapper
>> at java.net.URLClassLoader$1.run(**URLClassLoader.java:366)
>> at java.net.URLClassLoader$1.run(**URLClassLoader.java:355)
>> at java.security.**AccessController.doPrivileged(**Native Method)
>> at java.net.URLClassLoader.**findClass(URLClassLoader.java:**354)
>> at java.lang.ClassLoader.**loadClass(ClassLoader.java:**424)
>> at sun.misc.Launcher$**AppClassLoader.loadClass(**
>> Launcher.java:308)
>> at java.lang.ClassLoader.**loadClass(ClassLoader.java:**357)
>> at java.lang.Class.forName0(**Native Method)
>> at java.lang.Class.forName(Class.**java:270)
>> at org.apache.hadoop.conf.**Configuration.getClassByName(**
>> Configuration.java:820)
>> at org.apache.hadoop.conf.**Configuration.getClass(**
>> Configuration.java:865)
>> ... 8 more
>>
>>
>> But If I run same command from n2
>> [hduser@n2 ~]$ hadoop jar mahout-distribution-0.7/**mahout-examples-0.7-job.jar
>> org.apache.mahout.classifier.**df.mapreduce.BuildForest
>> -Dmapred.max.split.size=**1874231 -p -d testdata/bal_ee_2009.csv -ds
>> testdata/bal_ee_2009.csv.info -sl 10 -o bal_ee_2009_out -t 1
>>
>> 13/07/11 15:41:20 INFO mapreduce.BuildForest: Partial Mapred
>> implementation
>> 13/07/11 15:41:20 INFO mapreduce.BuildForest: Building the forest...
>> 13/07/11 15:41:21 INFO input.FileInputFormat: Total input paths to
>> process : 1
>> 13/07/11 15:41:21 INFO util.NativeCodeLoader: Loaded the native-hadoop
>> library
>> 13/07/11 15:41:21 WARN snappy.LoadSnappy: Snappy native library not loaded
>> 13/07/11 15:41:21 INFO mapred.JobClient: Running job:
>> job_201307111355_0017
>> 13/07/11 15:41:22 INFO mapred.JobClient: map 0% reduce 0%
>> 13/07/11 15:41:43 INFO mapred.JobClient: map 7% reduce 0%
>> 13/07/11 15:41:46 INFO mapred.JobClient: map 15% reduce 0%
>> 13/07/11 15:41:49 INFO mapred.JobClient: map 38% reduce 0%
>> 13/07/11 15:41:52 INFO mapred.JobClient: map 46% reduce 0%
>> 13/07/11 15:42:01 INFO mapred.JobClient: map 61% reduce 0%
>> 13/07/11 15:42:02 INFO mapred.JobClient: map 76% reduce 0%
>> 13/07/11 15:42:09 INFO mapred.JobClient: map 84% reduce 0%
>> 13/07/11 15:42:13 INFO mapred.JobClient: map 100% reduce 0%
>> 13/07/11 15:42:19 INFO mapred.JobClient: Job complete:
>> job_201307111355_0017
>> 13/07/11 15:42:19 INFO mapred.JobClient: Counters: 20
>> 13/07/11 15:42:19 INFO mapred.JobClient: Job Counters
>> 13/07/11 15:42:19 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=138911
>> 13/07/11 15:42:19 INFO mapred.JobClient: Total time spent by all
>> reduces waiting after reserving slots (ms)=0
>> 13/07/11 15:42:19 INFO mapred.JobClient: Total time spent by all maps
>> waiting after reserving slots (ms)=0
>> 13/07/11 15:42:19 INFO mapred.JobClient: Rack-local map tasks=6
>> 13/07/11 15:42:19 INFO mapred.JobClient: Launched map tasks=13
>> 13/07/11 15:42:19 INFO mapred.JobClient: Data-local map tasks=7
>> 13/07/11 15:42:19 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
>> 13/07/11 15:42:19 INFO mapred.JobClient: File Output Format Counters
>> 13/07/11 15:42:19 INFO mapred.JobClient: Bytes Written=184956
>> 13/07/11 15:42:19 INFO mapred.JobClient: FileSystemCounters
>> 13/07/11 15:42:19 INFO mapred.JobClient: HDFS_BYTES_READ=23444612
>> 13/07/11 15:42:19 INFO mapred.JobClient: FILE_BYTES_WRITTEN=296026
>> 13/07/11 15:42:19 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=184956
>> 13/07/11 15:42:19 INFO mapred.JobClient: File Input Format Counters
>> 13/07/11 15:42:19 INFO mapred.JobClient: Bytes Read=22952913
>> 13/07/11 15:42:19 INFO mapred.JobClient: Map-Reduce Framework
>> 13/07/11 15:42:19 INFO mapred.JobClient: Map input records=204091
>> 13/07/11 15:42:19 INFO mapred.JobClient: Physical memory (bytes)
>> snapshot=1370255360
>> 13/07/11 15:42:19 INFO mapred.JobClient: Spilled Records=0
>> 13/07/11 15:42:19 INFO mapred.JobClient: CPU time spent (ms)=27990
>> 13/07/11 15:42:19 INFO mapred.JobClient: Total committed heap usage
>> (bytes)=1292500992
>> 13/07/11 15:42:19 INFO mapred.JobClient: Virtual memory (bytes)
>> snapshot=37360082944
>> 13/07/11 15:42:19 INFO mapred.JobClient: Map output records=1
>> 13/07/11 15:42:19 INFO mapred.JobClient: SPLIT_RAW_BYTES=1573
>> 13/07/11 15:42:20 INFO common.HadoopUtil: Deleting
>> hdfs://master:54310/user/**hduser/bal_ee_2009_out
>> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Build Time: 0h 0m 59s 386
>> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Forest num Nodes: 10698
>> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Forest mean num Nodes: 10698
>> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Forest mean max Depth: 16
>> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Storing the forest in:
>> bal_ee_2009_out/forest.seq
>>
>> Both (n1 and n2) are used and from web console I can see that there are
>> no errors.
>>
>> Is there any explanations why I am getting errors when I run command from
>> master?
>>
>>
>> --
>> Regards, Margus (Margusja) Roo
>> +372 51 48 780
>> http://margus.roo.ee
>> skype: margusja
>> -----BEGIN PUBLIC KEY-----
>> MIGfMA0GCSqGSIb3DQEBAQUAA4GNAD**CBiQKBgQCvbeg7LwEC2SCpAEewwpC3**ajxE
>> 5ZsRMCB77L8bae9G7TslgLkoIzo9yO**jPdx2NN6DllKbV65UjTay43uUDyql9**g3tl
>> RhiJIcoAExkSTykWqAIPR88LfilLy1**JlQ+0RD8OXiWOVVQfhOHpQ0R/**jcAkM2lZa
>> BjM8j36yJvoBVsfOHQIDAQAB
>> -----END PUBLIC KEY-----
>>
>>
Re: Task failure in slave node
Posted by Azuryy Yu <az...@gmail.com>.
hi,
put all mahou jars under hadoop_home/lib, then restart cluster.
On Jul 11, 2013 8:45 PM, "Margusja" <ma...@roo.ee> wrote:
> Hi
>
> I have tow nodes:
> n1 (master, salve) and n2 (slave)
>
> after set up I ran wordcount example and it worked fine:
> [hduser@n1 ~]$ hadoop jar /usr/local/hadoop/hadoop-**examples-1.0.4.jar
> wordcount /user/hduser/gutenberg /user/hduser/gutenberg-output
> 13/07/11 15:30:44 INFO input.FileInputFormat: Total input paths to process
> : 7
> 13/07/11 15:30:44 INFO util.NativeCodeLoader: Loaded the native-hadoop
> library
> 13/07/11 15:30:44 WARN snappy.LoadSnappy: Snappy native library not loaded
> 13/07/11 15:30:44 INFO mapred.JobClient: Running job: job_201307111355_0015
> 13/07/11 15:30:45 INFO mapred.JobClient: map 0% reduce 0%
> 13/07/11 15:31:03 INFO mapred.JobClient: map 42% reduce 0%
> 13/07/11 15:31:06 INFO mapred.JobClient: map 57% reduce 0%
> 13/07/11 15:31:09 INFO mapred.JobClient: map 71% reduce 0%
> 13/07/11 15:31:15 INFO mapred.JobClient: map 100% reduce 0%
> 13/07/11 15:31:18 INFO mapred.JobClient: map 100% reduce 23%
> 13/07/11 15:31:27 INFO mapred.JobClient: map 100% reduce 100%
> 13/07/11 15:31:32 INFO mapred.JobClient: Job complete:
> job_201307111355_0015
> 13/07/11 15:31:32 INFO mapred.JobClient: Counters: 30
> 13/07/11 15:31:32 INFO mapred.JobClient: Job Counters
> 13/07/11 15:31:32 INFO mapred.JobClient: Launched reduce tasks=1
> 13/07/11 15:31:32 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=67576
> 13/07/11 15:31:32 INFO mapred.JobClient: Total time spent by all
> reduces waiting after reserving slots (ms)=0
> 13/07/11 15:31:32 INFO mapred.JobClient: Total time spent by all maps
> waiting after reserving slots (ms)=0
> 13/07/11 15:31:32 INFO mapred.JobClient: Rack-local map tasks=3
> 13/07/11 15:31:32 INFO mapred.JobClient: Launched map tasks=7
> 13/07/11 15:31:32 INFO mapred.JobClient: Data-local map tasks=4
> 13/07/11 15:31:32 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=21992
> 13/07/11 15:31:32 INFO mapred.JobClient: File Output Format Counters
> 13/07/11 15:31:32 INFO mapred.JobClient: Bytes Written=1412505
> 13/07/11 15:31:32 INFO mapred.JobClient: FileSystemCounters
> 13/07/11 15:31:32 INFO mapred.JobClient: FILE_BYTES_READ=5414195
> 13/07/11 15:31:32 INFO mapred.JobClient: HDFS_BYTES_READ=6950820
> 13/07/11 15:31:32 INFO mapred.JobClient: FILE_BYTES_WRITTEN=8744993
> 13/07/11 15:31:32 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=1412505
> 13/07/11 15:31:32 INFO mapred.JobClient: File Input Format Counters
> 13/07/11 15:31:32 INFO mapred.JobClient: Bytes Read=6950001
> 13/07/11 15:31:32 INFO mapred.JobClient: Map-Reduce Framework
> 13/07/11 15:31:32 INFO mapred.JobClient: Map output materialized
> bytes=3157469
> 13/07/11 15:31:32 INFO mapred.JobClient: Map input records=137146
> 13/07/11 15:31:32 INFO mapred.JobClient: Reduce shuffle bytes=2904836
> 13/07/11 15:31:32 INFO mapred.JobClient: Spilled Records=594764
> 13/07/11 15:31:32 INFO mapred.JobClient: Map output bytes=11435849
> 13/07/11 15:31:32 INFO mapred.JobClient: Total committed heap usage
> (bytes)=1128136704
> 13/07/11 15:31:32 INFO mapred.JobClient: CPU time spent (ms)=18230
> 13/07/11 15:31:32 INFO mapred.JobClient: Combine input records=1174991
> 13/07/11 15:31:32 INFO mapred.JobClient: SPLIT_RAW_BYTES=819
> 13/07/11 15:31:32 INFO mapred.JobClient: Reduce input records=218990
> 13/07/11 15:31:32 INFO mapred.JobClient: Reduce input groups=128513
> 13/07/11 15:31:32 INFO mapred.JobClient: Combine output records=218990
> 13/07/11 15:31:32 INFO mapred.JobClient: Physical memory (bytes)
> snapshot=1179656192
> 13/07/11 15:31:32 INFO mapred.JobClient: Reduce output records=128513
> 13/07/11 15:31:32 INFO mapred.JobClient: Virtual memory (bytes)
> snapshot=22992117760
> 13/07/11 15:31:32 INFO mapred.JobClient: Map output records=1174991
>
> from web interface (http://n1:50030/) I saw that both (n1 and n2 ) were
> used without any errors.
>
> Problems appear if I try to use following commands in master (n1):
>
> [hduser@n1 ~]$hadoop jar mahout-distribution-0.7/**mahout-examples-0.7-job.jar
> org.apache.mahout.classifier.**df.mapreduce.BuildForest
> -Dmapred.max.split.size=**1874231 -p -d testdata/bal_ee_2009.csv -ds
> testdata/bal_ee_2009.csv.info -sl 10 -o bal_ee_2009_out -t 1
>
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in [file:/usr/local/hadoop-1.0.4/**org/slf4j/impl/**
> StaticLoggerBinder.class]
> SLF4J: Found binding in [jar:file:/usr/local/hadoop-1.**
> 0.4/lib/slf4j-log4j12-1.4.3.**jar!/org/slf4j/impl/**
> StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.**html#multiple_bindings<http://www.slf4j.org/codes.html#multiple_bindings>for an explanation.
> 13/07/11 15:36:50 INFO mapreduce.BuildForest: Partial Mapred implementation
> 13/07/11 15:36:50 INFO mapreduce.BuildForest: Building the forest...
> 13/07/11 15:36:50 WARN mapred.JobClient: No job jar file set. User
> classes may not be found. See JobConf(Class) or JobConf#setJar(String).
> 13/07/11 15:36:50 INFO input.FileInputFormat: Total input paths to process
> : 1
> 13/07/11 15:36:50 INFO util.NativeCodeLoader: Loaded the native-hadoop
> library
> 13/07/11 15:36:50 WARN snappy.LoadSnappy: Snappy native library not loaded
> 13/07/11 15:36:50 INFO mapred.JobClient: Running job: job_201307111355_0016
> 13/07/11 15:36:51 INFO mapred.JobClient: map 0% reduce 0%
> 13/07/11 15:37:09 INFO mapred.JobClient: map 7% reduce 0%
> 13/07/11 15:37:11 INFO mapred.JobClient: Task Id :
> attempt_201307111355_0016_m_**000002_0, Status : FAILED
> java.lang.RuntimeException: java.lang.**ClassNotFoundException:
> org.apache.mahout.classifier.**df.mapreduce.partial.**Step1Mapper
> at org.apache.hadoop.conf.**Configuration.getClass(**
> Configuration.java:867)
> at org.apache.hadoop.mapreduce.**JobContext.getMapperClass(**
> JobContext.java:199)
> at org.apache.hadoop.mapred.**MapTask.runNewMapper(MapTask.**
> java:719)
> at org.apache.hadoop.mapred.**MapTask.run(MapTask.java:370)
> at org.apache.hadoop.mapred.**Child$4.run(Child.java:255)
> at java.security.**AccessController.doPrivileged(**Native Method)
> at javax.security.auth.Subject.**doAs(Subject.java:415)
> at org.apache.hadoop.security.**UserGroupInformation.doAs(**
> UserGroupInformation.java:**1121)
> at org.apache.hadoop.mapred.**Child.main(Child.java:249)
> Caused by: java.lang.**ClassNotFoundException:
> org.apache.mahout.classifier.**df.mapreduce.partial.**Step1Mapper
> at java.net.URLClassLoader$1.run(**URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(**URLClassLoader.java:355)
> at java.security.**AccessController.doPrivileged(**Native Method)
> at java.net.URLClassLoader.**findClass(URLClassLoader.java:**354)
> at java.lang.ClassLoader.**loadClass(ClassLoader.java:**424)
> at sun.misc.Launcher$**AppClassLoader.loadClass(**
> Launcher.java:308)
> at java.lang.ClassLoader.**loadClass(ClassLoader.java:**357)
> at java.lang.Class.forName0(**Native Method)
> at java.lang.Class.forName(Class.**java:270)
> at org.apache.hadoop.conf.**Configuration.getClassByName(**
> Configuration.java:820)
> at org.apache.hadoop.conf.**Configuration.getClass(**
> Configuration.java:865)
> ... 8 more
>
> ...
> ...
> ...
>
> 13/07/11 15:37:37 INFO mapred.JobClient: map 61% reduce 0%
> 13/07/11 15:37:46 INFO mapred.JobClient: map 76% reduce 0%
> 13/07/11 15:37:55 INFO mapred.JobClient: map 92% reduce 0%
> 13/07/11 15:38:01 INFO mapred.JobClient: map 100% reduce 0%
> 13/07/11 15:38:06 INFO mapred.JobClient: Job complete:
> job_201307111355_0016
> 13/07/11 15:38:06 INFO mapred.JobClient: Counters: 20
> 13/07/11 15:38:06 INFO mapred.JobClient: Job Counters
> 13/07/11 15:38:06 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=184724
> 13/07/11 15:38:06 INFO mapred.JobClient: Total time spent by all
> reduces waiting after reserving slots (ms)=0
> 13/07/11 15:38:06 INFO mapred.JobClient: Total time spent by all maps
> waiting after reserving slots (ms)=0
> 13/07/11 15:38:06 INFO mapred.JobClient: Rack-local map tasks=13
> 13/07/11 15:38:06 INFO mapred.JobClient: Launched map tasks=22
> 13/07/11 15:38:06 INFO mapred.JobClient: Data-local map tasks=9
> 13/07/11 15:38:06 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
> 13/07/11 15:38:06 INFO mapred.JobClient: File Output Format Counters
> 13/07/11 15:38:06 INFO mapred.JobClient: Bytes Written=184520
> 13/07/11 15:38:06 INFO mapred.JobClient: FileSystemCounters
> 13/07/11 15:38:06 INFO mapred.JobClient: HDFS_BYTES_READ=23444612
> 13/07/11 15:38:06 INFO mapred.JobClient: FILE_BYTES_WRITTEN=294037
> 13/07/11 15:38:06 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=184520
> 13/07/11 15:38:06 INFO mapred.JobClient: File Input Format Counters
> 13/07/11 15:38:06 INFO mapred.JobClient: Bytes Read=22952913
> 13/07/11 15:38:06 INFO mapred.JobClient: Map-Reduce Framework
> 13/07/11 15:38:06 INFO mapred.JobClient: Map input records=204091
> 13/07/11 15:38:06 INFO mapred.JobClient: Physical memory (bytes)
> snapshot=1402363904
> 13/07/11 15:38:06 INFO mapred.JobClient: Spilled Records=0
> 13/07/11 15:38:06 INFO mapred.JobClient: CPU time spent (ms)=26640
> 13/07/11 15:38:06 INFO mapred.JobClient: Total committed heap usage
> (bytes)=1333985280
> 13/07/11 15:38:06 INFO mapred.JobClient: Virtual memory (bytes)
> snapshot=37349126144
> 13/07/11 15:38:06 INFO mapred.JobClient: Map output records=1
> 13/07/11 15:38:06 INFO mapred.JobClient: SPLIT_RAW_BYTES=1573
> 13/07/11 15:38:07 INFO common.HadoopUtil: Deleting
> hdfs://master:54310/user/**hduser/bal_ee_2009_out
> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Build Time: 0h 1m 16s 977
> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Forest num Nodes: 10700
> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Forest mean num Nodes: 10700
> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Forest mean max Depth: 17
> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Storing the forest in:
> bal_ee_2009_out/forest.seq
>
> From web console I can see that n1 tasks are fine but n2 tasks failed:
>
> java.lang.RuntimeException: java.lang.**ClassNotFoundException:
> org.apache.mahout.classifier.**df.mapreduce.partial.**Step1Mapper
> at org.apache.hadoop.conf.**Configuration.getClass(**
> Configuration.java:867)
> at org.apache.hadoop.mapreduce.**JobContext.getMapperClass(**
> JobContext.java:199)
> at org.apache.hadoop.mapred.**MapTask.runNewMapper(MapTask.**
> java:719)
> at org.apache.hadoop.mapred.**MapTask.run(MapTask.java:370)
> at org.apache.hadoop.mapred.**Child$4.run(Child.java:255)
> at java.security.**AccessController.doPrivileged(**Native Method)
> at javax.security.auth.Subject.**doAs(Subject.java:415)
> at org.apache.hadoop.security.**UserGroupInformation.doAs(**
> UserGroupInformation.java:**1121)
> at org.apache.hadoop.mapred.**Child.main(Child.java:249)
> Caused by: java.lang.**ClassNotFoundException:
> org.apache.mahout.classifier.**df.mapreduce.partial.**Step1Mapper
> at java.net.URLClassLoader$1.run(**URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(**URLClassLoader.java:355)
> at java.security.**AccessController.doPrivileged(**Native Method)
> at java.net.URLClassLoader.**findClass(URLClassLoader.java:**354)
> at java.lang.ClassLoader.**loadClass(ClassLoader.java:**424)
> at sun.misc.Launcher$**AppClassLoader.loadClass(**
> Launcher.java:308)
> at java.lang.ClassLoader.**loadClass(ClassLoader.java:**357)
> at java.lang.Class.forName0(**Native Method)
> at java.lang.Class.forName(Class.**java:270)
> at org.apache.hadoop.conf.**Configuration.getClassByName(**
> Configuration.java:820)
> at org.apache.hadoop.conf.**Configuration.getClass(**
> Configuration.java:865)
> ... 8 more
>
>
> But If I run same command from n2
> [hduser@n2 ~]$ hadoop jar mahout-distribution-0.7/**mahout-examples-0.7-job.jar
> org.apache.mahout.classifier.**df.mapreduce.BuildForest
> -Dmapred.max.split.size=**1874231 -p -d testdata/bal_ee_2009.csv -ds
> testdata/bal_ee_2009.csv.info -sl 10 -o bal_ee_2009_out -t 1
>
> 13/07/11 15:41:20 INFO mapreduce.BuildForest: Partial Mapred implementation
> 13/07/11 15:41:20 INFO mapreduce.BuildForest: Building the forest...
> 13/07/11 15:41:21 INFO input.FileInputFormat: Total input paths to process
> : 1
> 13/07/11 15:41:21 INFO util.NativeCodeLoader: Loaded the native-hadoop
> library
> 13/07/11 15:41:21 WARN snappy.LoadSnappy: Snappy native library not loaded
> 13/07/11 15:41:21 INFO mapred.JobClient: Running job: job_201307111355_0017
> 13/07/11 15:41:22 INFO mapred.JobClient: map 0% reduce 0%
> 13/07/11 15:41:43 INFO mapred.JobClient: map 7% reduce 0%
> 13/07/11 15:41:46 INFO mapred.JobClient: map 15% reduce 0%
> 13/07/11 15:41:49 INFO mapred.JobClient: map 38% reduce 0%
> 13/07/11 15:41:52 INFO mapred.JobClient: map 46% reduce 0%
> 13/07/11 15:42:01 INFO mapred.JobClient: map 61% reduce 0%
> 13/07/11 15:42:02 INFO mapred.JobClient: map 76% reduce 0%
> 13/07/11 15:42:09 INFO mapred.JobClient: map 84% reduce 0%
> 13/07/11 15:42:13 INFO mapred.JobClient: map 100% reduce 0%
> 13/07/11 15:42:19 INFO mapred.JobClient: Job complete:
> job_201307111355_0017
> 13/07/11 15:42:19 INFO mapred.JobClient: Counters: 20
> 13/07/11 15:42:19 INFO mapred.JobClient: Job Counters
> 13/07/11 15:42:19 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=138911
> 13/07/11 15:42:19 INFO mapred.JobClient: Total time spent by all
> reduces waiting after reserving slots (ms)=0
> 13/07/11 15:42:19 INFO mapred.JobClient: Total time spent by all maps
> waiting after reserving slots (ms)=0
> 13/07/11 15:42:19 INFO mapred.JobClient: Rack-local map tasks=6
> 13/07/11 15:42:19 INFO mapred.JobClient: Launched map tasks=13
> 13/07/11 15:42:19 INFO mapred.JobClient: Data-local map tasks=7
> 13/07/11 15:42:19 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
> 13/07/11 15:42:19 INFO mapred.JobClient: File Output Format Counters
> 13/07/11 15:42:19 INFO mapred.JobClient: Bytes Written=184956
> 13/07/11 15:42:19 INFO mapred.JobClient: FileSystemCounters
> 13/07/11 15:42:19 INFO mapred.JobClient: HDFS_BYTES_READ=23444612
> 13/07/11 15:42:19 INFO mapred.JobClient: FILE_BYTES_WRITTEN=296026
> 13/07/11 15:42:19 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=184956
> 13/07/11 15:42:19 INFO mapred.JobClient: File Input Format Counters
> 13/07/11 15:42:19 INFO mapred.JobClient: Bytes Read=22952913
> 13/07/11 15:42:19 INFO mapred.JobClient: Map-Reduce Framework
> 13/07/11 15:42:19 INFO mapred.JobClient: Map input records=204091
> 13/07/11 15:42:19 INFO mapred.JobClient: Physical memory (bytes)
> snapshot=1370255360
> 13/07/11 15:42:19 INFO mapred.JobClient: Spilled Records=0
> 13/07/11 15:42:19 INFO mapred.JobClient: CPU time spent (ms)=27990
> 13/07/11 15:42:19 INFO mapred.JobClient: Total committed heap usage
> (bytes)=1292500992
> 13/07/11 15:42:19 INFO mapred.JobClient: Virtual memory (bytes)
> snapshot=37360082944
> 13/07/11 15:42:19 INFO mapred.JobClient: Map output records=1
> 13/07/11 15:42:19 INFO mapred.JobClient: SPLIT_RAW_BYTES=1573
> 13/07/11 15:42:20 INFO common.HadoopUtil: Deleting
> hdfs://master:54310/user/**hduser/bal_ee_2009_out
> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Build Time: 0h 0m 59s 386
> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Forest num Nodes: 10698
> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Forest mean num Nodes: 10698
> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Forest mean max Depth: 16
> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Storing the forest in:
> bal_ee_2009_out/forest.seq
>
> Both (n1 and n2) are used and from web console I can see that there are no
> errors.
>
> Is there any explanations why I am getting errors when I run command from
> master?
>
>
> --
> Regards, Margus (Margusja) Roo
> +372 51 48 780
> http://margus.roo.ee
> skype: margusja
> -----BEGIN PUBLIC KEY-----
> MIGfMA0GCSqGSIb3DQEBAQUAA4GNAD**CBiQKBgQCvbeg7LwEC2SCpAEewwpC3**ajxE
> 5ZsRMCB77L8bae9G7TslgLkoIzo9yO**jPdx2NN6DllKbV65UjTay43uUDyql9**g3tl
> RhiJIcoAExkSTykWqAIPR88LfilLy1**JlQ+0RD8OXiWOVVQfhOHpQ0R/**jcAkM2lZa
> BjM8j36yJvoBVsfOHQIDAQAB
> -----END PUBLIC KEY-----
>
>
Re: Task failure in slave node
Posted by Azuryy Yu <az...@gmail.com>.
hi,
put all mahou jars under hadoop_home/lib, then restart cluster.
On Jul 11, 2013 8:45 PM, "Margusja" <ma...@roo.ee> wrote:
> Hi
>
> I have tow nodes:
> n1 (master, salve) and n2 (slave)
>
> after set up I ran wordcount example and it worked fine:
> [hduser@n1 ~]$ hadoop jar /usr/local/hadoop/hadoop-**examples-1.0.4.jar
> wordcount /user/hduser/gutenberg /user/hduser/gutenberg-output
> 13/07/11 15:30:44 INFO input.FileInputFormat: Total input paths to process
> : 7
> 13/07/11 15:30:44 INFO util.NativeCodeLoader: Loaded the native-hadoop
> library
> 13/07/11 15:30:44 WARN snappy.LoadSnappy: Snappy native library not loaded
> 13/07/11 15:30:44 INFO mapred.JobClient: Running job: job_201307111355_0015
> 13/07/11 15:30:45 INFO mapred.JobClient: map 0% reduce 0%
> 13/07/11 15:31:03 INFO mapred.JobClient: map 42% reduce 0%
> 13/07/11 15:31:06 INFO mapred.JobClient: map 57% reduce 0%
> 13/07/11 15:31:09 INFO mapred.JobClient: map 71% reduce 0%
> 13/07/11 15:31:15 INFO mapred.JobClient: map 100% reduce 0%
> 13/07/11 15:31:18 INFO mapred.JobClient: map 100% reduce 23%
> 13/07/11 15:31:27 INFO mapred.JobClient: map 100% reduce 100%
> 13/07/11 15:31:32 INFO mapred.JobClient: Job complete:
> job_201307111355_0015
> 13/07/11 15:31:32 INFO mapred.JobClient: Counters: 30
> 13/07/11 15:31:32 INFO mapred.JobClient: Job Counters
> 13/07/11 15:31:32 INFO mapred.JobClient: Launched reduce tasks=1
> 13/07/11 15:31:32 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=67576
> 13/07/11 15:31:32 INFO mapred.JobClient: Total time spent by all
> reduces waiting after reserving slots (ms)=0
> 13/07/11 15:31:32 INFO mapred.JobClient: Total time spent by all maps
> waiting after reserving slots (ms)=0
> 13/07/11 15:31:32 INFO mapred.JobClient: Rack-local map tasks=3
> 13/07/11 15:31:32 INFO mapred.JobClient: Launched map tasks=7
> 13/07/11 15:31:32 INFO mapred.JobClient: Data-local map tasks=4
> 13/07/11 15:31:32 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=21992
> 13/07/11 15:31:32 INFO mapred.JobClient: File Output Format Counters
> 13/07/11 15:31:32 INFO mapred.JobClient: Bytes Written=1412505
> 13/07/11 15:31:32 INFO mapred.JobClient: FileSystemCounters
> 13/07/11 15:31:32 INFO mapred.JobClient: FILE_BYTES_READ=5414195
> 13/07/11 15:31:32 INFO mapred.JobClient: HDFS_BYTES_READ=6950820
> 13/07/11 15:31:32 INFO mapred.JobClient: FILE_BYTES_WRITTEN=8744993
> 13/07/11 15:31:32 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=1412505
> 13/07/11 15:31:32 INFO mapred.JobClient: File Input Format Counters
> 13/07/11 15:31:32 INFO mapred.JobClient: Bytes Read=6950001
> 13/07/11 15:31:32 INFO mapred.JobClient: Map-Reduce Framework
> 13/07/11 15:31:32 INFO mapred.JobClient: Map output materialized
> bytes=3157469
> 13/07/11 15:31:32 INFO mapred.JobClient: Map input records=137146
> 13/07/11 15:31:32 INFO mapred.JobClient: Reduce shuffle bytes=2904836
> 13/07/11 15:31:32 INFO mapred.JobClient: Spilled Records=594764
> 13/07/11 15:31:32 INFO mapred.JobClient: Map output bytes=11435849
> 13/07/11 15:31:32 INFO mapred.JobClient: Total committed heap usage
> (bytes)=1128136704
> 13/07/11 15:31:32 INFO mapred.JobClient: CPU time spent (ms)=18230
> 13/07/11 15:31:32 INFO mapred.JobClient: Combine input records=1174991
> 13/07/11 15:31:32 INFO mapred.JobClient: SPLIT_RAW_BYTES=819
> 13/07/11 15:31:32 INFO mapred.JobClient: Reduce input records=218990
> 13/07/11 15:31:32 INFO mapred.JobClient: Reduce input groups=128513
> 13/07/11 15:31:32 INFO mapred.JobClient: Combine output records=218990
> 13/07/11 15:31:32 INFO mapred.JobClient: Physical memory (bytes)
> snapshot=1179656192
> 13/07/11 15:31:32 INFO mapred.JobClient: Reduce output records=128513
> 13/07/11 15:31:32 INFO mapred.JobClient: Virtual memory (bytes)
> snapshot=22992117760
> 13/07/11 15:31:32 INFO mapred.JobClient: Map output records=1174991
>
> from web interface (http://n1:50030/) I saw that both (n1 and n2 ) were
> used without any errors.
>
> Problems appear if I try to use following commands in master (n1):
>
> [hduser@n1 ~]$hadoop jar mahout-distribution-0.7/**mahout-examples-0.7-job.jar
> org.apache.mahout.classifier.**df.mapreduce.BuildForest
> -Dmapred.max.split.size=**1874231 -p -d testdata/bal_ee_2009.csv -ds
> testdata/bal_ee_2009.csv.info -sl 10 -o bal_ee_2009_out -t 1
>
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in [file:/usr/local/hadoop-1.0.4/**org/slf4j/impl/**
> StaticLoggerBinder.class]
> SLF4J: Found binding in [jar:file:/usr/local/hadoop-1.**
> 0.4/lib/slf4j-log4j12-1.4.3.**jar!/org/slf4j/impl/**
> StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.**html#multiple_bindings<http://www.slf4j.org/codes.html#multiple_bindings>for an explanation.
> 13/07/11 15:36:50 INFO mapreduce.BuildForest: Partial Mapred implementation
> 13/07/11 15:36:50 INFO mapreduce.BuildForest: Building the forest...
> 13/07/11 15:36:50 WARN mapred.JobClient: No job jar file set. User
> classes may not be found. See JobConf(Class) or JobConf#setJar(String).
> 13/07/11 15:36:50 INFO input.FileInputFormat: Total input paths to process
> : 1
> 13/07/11 15:36:50 INFO util.NativeCodeLoader: Loaded the native-hadoop
> library
> 13/07/11 15:36:50 WARN snappy.LoadSnappy: Snappy native library not loaded
> 13/07/11 15:36:50 INFO mapred.JobClient: Running job: job_201307111355_0016
> 13/07/11 15:36:51 INFO mapred.JobClient: map 0% reduce 0%
> 13/07/11 15:37:09 INFO mapred.JobClient: map 7% reduce 0%
> 13/07/11 15:37:11 INFO mapred.JobClient: Task Id :
> attempt_201307111355_0016_m_**000002_0, Status : FAILED
> java.lang.RuntimeException: java.lang.**ClassNotFoundException:
> org.apache.mahout.classifier.**df.mapreduce.partial.**Step1Mapper
> at org.apache.hadoop.conf.**Configuration.getClass(**
> Configuration.java:867)
> at org.apache.hadoop.mapreduce.**JobContext.getMapperClass(**
> JobContext.java:199)
> at org.apache.hadoop.mapred.**MapTask.runNewMapper(MapTask.**
> java:719)
> at org.apache.hadoop.mapred.**MapTask.run(MapTask.java:370)
> at org.apache.hadoop.mapred.**Child$4.run(Child.java:255)
> at java.security.**AccessController.doPrivileged(**Native Method)
> at javax.security.auth.Subject.**doAs(Subject.java:415)
> at org.apache.hadoop.security.**UserGroupInformation.doAs(**
> UserGroupInformation.java:**1121)
> at org.apache.hadoop.mapred.**Child.main(Child.java:249)
> Caused by: java.lang.**ClassNotFoundException:
> org.apache.mahout.classifier.**df.mapreduce.partial.**Step1Mapper
> at java.net.URLClassLoader$1.run(**URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(**URLClassLoader.java:355)
> at java.security.**AccessController.doPrivileged(**Native Method)
> at java.net.URLClassLoader.**findClass(URLClassLoader.java:**354)
> at java.lang.ClassLoader.**loadClass(ClassLoader.java:**424)
> at sun.misc.Launcher$**AppClassLoader.loadClass(**
> Launcher.java:308)
> at java.lang.ClassLoader.**loadClass(ClassLoader.java:**357)
> at java.lang.Class.forName0(**Native Method)
> at java.lang.Class.forName(Class.**java:270)
> at org.apache.hadoop.conf.**Configuration.getClassByName(**
> Configuration.java:820)
> at org.apache.hadoop.conf.**Configuration.getClass(**
> Configuration.java:865)
> ... 8 more
>
> ...
> ...
> ...
>
> 13/07/11 15:37:37 INFO mapred.JobClient: map 61% reduce 0%
> 13/07/11 15:37:46 INFO mapred.JobClient: map 76% reduce 0%
> 13/07/11 15:37:55 INFO mapred.JobClient: map 92% reduce 0%
> 13/07/11 15:38:01 INFO mapred.JobClient: map 100% reduce 0%
> 13/07/11 15:38:06 INFO mapred.JobClient: Job complete:
> job_201307111355_0016
> 13/07/11 15:38:06 INFO mapred.JobClient: Counters: 20
> 13/07/11 15:38:06 INFO mapred.JobClient: Job Counters
> 13/07/11 15:38:06 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=184724
> 13/07/11 15:38:06 INFO mapred.JobClient: Total time spent by all
> reduces waiting after reserving slots (ms)=0
> 13/07/11 15:38:06 INFO mapred.JobClient: Total time spent by all maps
> waiting after reserving slots (ms)=0
> 13/07/11 15:38:06 INFO mapred.JobClient: Rack-local map tasks=13
> 13/07/11 15:38:06 INFO mapred.JobClient: Launched map tasks=22
> 13/07/11 15:38:06 INFO mapred.JobClient: Data-local map tasks=9
> 13/07/11 15:38:06 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
> 13/07/11 15:38:06 INFO mapred.JobClient: File Output Format Counters
> 13/07/11 15:38:06 INFO mapred.JobClient: Bytes Written=184520
> 13/07/11 15:38:06 INFO mapred.JobClient: FileSystemCounters
> 13/07/11 15:38:06 INFO mapred.JobClient: HDFS_BYTES_READ=23444612
> 13/07/11 15:38:06 INFO mapred.JobClient: FILE_BYTES_WRITTEN=294037
> 13/07/11 15:38:06 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=184520
> 13/07/11 15:38:06 INFO mapred.JobClient: File Input Format Counters
> 13/07/11 15:38:06 INFO mapred.JobClient: Bytes Read=22952913
> 13/07/11 15:38:06 INFO mapred.JobClient: Map-Reduce Framework
> 13/07/11 15:38:06 INFO mapred.JobClient: Map input records=204091
> 13/07/11 15:38:06 INFO mapred.JobClient: Physical memory (bytes)
> snapshot=1402363904
> 13/07/11 15:38:06 INFO mapred.JobClient: Spilled Records=0
> 13/07/11 15:38:06 INFO mapred.JobClient: CPU time spent (ms)=26640
> 13/07/11 15:38:06 INFO mapred.JobClient: Total committed heap usage
> (bytes)=1333985280
> 13/07/11 15:38:06 INFO mapred.JobClient: Virtual memory (bytes)
> snapshot=37349126144
> 13/07/11 15:38:06 INFO mapred.JobClient: Map output records=1
> 13/07/11 15:38:06 INFO mapred.JobClient: SPLIT_RAW_BYTES=1573
> 13/07/11 15:38:07 INFO common.HadoopUtil: Deleting
> hdfs://master:54310/user/**hduser/bal_ee_2009_out
> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Build Time: 0h 1m 16s 977
> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Forest num Nodes: 10700
> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Forest mean num Nodes: 10700
> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Forest mean max Depth: 17
> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Storing the forest in:
> bal_ee_2009_out/forest.seq
>
> From web console I can see that n1 tasks are fine but n2 tasks failed:
>
> java.lang.RuntimeException: java.lang.**ClassNotFoundException:
> org.apache.mahout.classifier.**df.mapreduce.partial.**Step1Mapper
> at org.apache.hadoop.conf.**Configuration.getClass(**
> Configuration.java:867)
> at org.apache.hadoop.mapreduce.**JobContext.getMapperClass(**
> JobContext.java:199)
> at org.apache.hadoop.mapred.**MapTask.runNewMapper(MapTask.**
> java:719)
> at org.apache.hadoop.mapred.**MapTask.run(MapTask.java:370)
> at org.apache.hadoop.mapred.**Child$4.run(Child.java:255)
> at java.security.**AccessController.doPrivileged(**Native Method)
> at javax.security.auth.Subject.**doAs(Subject.java:415)
> at org.apache.hadoop.security.**UserGroupInformation.doAs(**
> UserGroupInformation.java:**1121)
> at org.apache.hadoop.mapred.**Child.main(Child.java:249)
> Caused by: java.lang.**ClassNotFoundException:
> org.apache.mahout.classifier.**df.mapreduce.partial.**Step1Mapper
> at java.net.URLClassLoader$1.run(**URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(**URLClassLoader.java:355)
> at java.security.**AccessController.doPrivileged(**Native Method)
> at java.net.URLClassLoader.**findClass(URLClassLoader.java:**354)
> at java.lang.ClassLoader.**loadClass(ClassLoader.java:**424)
> at sun.misc.Launcher$**AppClassLoader.loadClass(**
> Launcher.java:308)
> at java.lang.ClassLoader.**loadClass(ClassLoader.java:**357)
> at java.lang.Class.forName0(**Native Method)
> at java.lang.Class.forName(Class.**java:270)
> at org.apache.hadoop.conf.**Configuration.getClassByName(**
> Configuration.java:820)
> at org.apache.hadoop.conf.**Configuration.getClass(**
> Configuration.java:865)
> ... 8 more
>
>
> But If I run same command from n2
> [hduser@n2 ~]$ hadoop jar mahout-distribution-0.7/**mahout-examples-0.7-job.jar
> org.apache.mahout.classifier.**df.mapreduce.BuildForest
> -Dmapred.max.split.size=**1874231 -p -d testdata/bal_ee_2009.csv -ds
> testdata/bal_ee_2009.csv.info -sl 10 -o bal_ee_2009_out -t 1
>
> 13/07/11 15:41:20 INFO mapreduce.BuildForest: Partial Mapred implementation
> 13/07/11 15:41:20 INFO mapreduce.BuildForest: Building the forest...
> 13/07/11 15:41:21 INFO input.FileInputFormat: Total input paths to process
> : 1
> 13/07/11 15:41:21 INFO util.NativeCodeLoader: Loaded the native-hadoop
> library
> 13/07/11 15:41:21 WARN snappy.LoadSnappy: Snappy native library not loaded
> 13/07/11 15:41:21 INFO mapred.JobClient: Running job: job_201307111355_0017
> 13/07/11 15:41:22 INFO mapred.JobClient: map 0% reduce 0%
> 13/07/11 15:41:43 INFO mapred.JobClient: map 7% reduce 0%
> 13/07/11 15:41:46 INFO mapred.JobClient: map 15% reduce 0%
> 13/07/11 15:41:49 INFO mapred.JobClient: map 38% reduce 0%
> 13/07/11 15:41:52 INFO mapred.JobClient: map 46% reduce 0%
> 13/07/11 15:42:01 INFO mapred.JobClient: map 61% reduce 0%
> 13/07/11 15:42:02 INFO mapred.JobClient: map 76% reduce 0%
> 13/07/11 15:42:09 INFO mapred.JobClient: map 84% reduce 0%
> 13/07/11 15:42:13 INFO mapred.JobClient: map 100% reduce 0%
> 13/07/11 15:42:19 INFO mapred.JobClient: Job complete:
> job_201307111355_0017
> 13/07/11 15:42:19 INFO mapred.JobClient: Counters: 20
> 13/07/11 15:42:19 INFO mapred.JobClient: Job Counters
> 13/07/11 15:42:19 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=138911
> 13/07/11 15:42:19 INFO mapred.JobClient: Total time spent by all
> reduces waiting after reserving slots (ms)=0
> 13/07/11 15:42:19 INFO mapred.JobClient: Total time spent by all maps
> waiting after reserving slots (ms)=0
> 13/07/11 15:42:19 INFO mapred.JobClient: Rack-local map tasks=6
> 13/07/11 15:42:19 INFO mapred.JobClient: Launched map tasks=13
> 13/07/11 15:42:19 INFO mapred.JobClient: Data-local map tasks=7
> 13/07/11 15:42:19 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
> 13/07/11 15:42:19 INFO mapred.JobClient: File Output Format Counters
> 13/07/11 15:42:19 INFO mapred.JobClient: Bytes Written=184956
> 13/07/11 15:42:19 INFO mapred.JobClient: FileSystemCounters
> 13/07/11 15:42:19 INFO mapred.JobClient: HDFS_BYTES_READ=23444612
> 13/07/11 15:42:19 INFO mapred.JobClient: FILE_BYTES_WRITTEN=296026
> 13/07/11 15:42:19 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=184956
> 13/07/11 15:42:19 INFO mapred.JobClient: File Input Format Counters
> 13/07/11 15:42:19 INFO mapred.JobClient: Bytes Read=22952913
> 13/07/11 15:42:19 INFO mapred.JobClient: Map-Reduce Framework
> 13/07/11 15:42:19 INFO mapred.JobClient: Map input records=204091
> 13/07/11 15:42:19 INFO mapred.JobClient: Physical memory (bytes)
> snapshot=1370255360
> 13/07/11 15:42:19 INFO mapred.JobClient: Spilled Records=0
> 13/07/11 15:42:19 INFO mapred.JobClient: CPU time spent (ms)=27990
> 13/07/11 15:42:19 INFO mapred.JobClient: Total committed heap usage
> (bytes)=1292500992
> 13/07/11 15:42:19 INFO mapred.JobClient: Virtual memory (bytes)
> snapshot=37360082944
> 13/07/11 15:42:19 INFO mapred.JobClient: Map output records=1
> 13/07/11 15:42:19 INFO mapred.JobClient: SPLIT_RAW_BYTES=1573
> 13/07/11 15:42:20 INFO common.HadoopUtil: Deleting
> hdfs://master:54310/user/**hduser/bal_ee_2009_out
> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Build Time: 0h 0m 59s 386
> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Forest num Nodes: 10698
> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Forest mean num Nodes: 10698
> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Forest mean max Depth: 16
> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Storing the forest in:
> bal_ee_2009_out/forest.seq
>
> Both (n1 and n2) are used and from web console I can see that there are no
> errors.
>
> Is there any explanations why I am getting errors when I run command from
> master?
>
>
> --
> Regards, Margus (Margusja) Roo
> +372 51 48 780
> http://margus.roo.ee
> skype: margusja
> -----BEGIN PUBLIC KEY-----
> MIGfMA0GCSqGSIb3DQEBAQUAA4GNAD**CBiQKBgQCvbeg7LwEC2SCpAEewwpC3**ajxE
> 5ZsRMCB77L8bae9G7TslgLkoIzo9yO**jPdx2NN6DllKbV65UjTay43uUDyql9**g3tl
> RhiJIcoAExkSTykWqAIPR88LfilLy1**JlQ+0RD8OXiWOVVQfhOHpQ0R/**jcAkM2lZa
> BjM8j36yJvoBVsfOHQIDAQAB
> -----END PUBLIC KEY-----
>
>
Re: Task failure in slave node
Posted by Azuryy Yu <az...@gmail.com>.
hi,
put all mahou jars under hadoop_home/lib, then restart cluster.
On Jul 11, 2013 8:45 PM, "Margusja" <ma...@roo.ee> wrote:
> Hi
>
> I have tow nodes:
> n1 (master, salve) and n2 (slave)
>
> after set up I ran wordcount example and it worked fine:
> [hduser@n1 ~]$ hadoop jar /usr/local/hadoop/hadoop-**examples-1.0.4.jar
> wordcount /user/hduser/gutenberg /user/hduser/gutenberg-output
> 13/07/11 15:30:44 INFO input.FileInputFormat: Total input paths to process
> : 7
> 13/07/11 15:30:44 INFO util.NativeCodeLoader: Loaded the native-hadoop
> library
> 13/07/11 15:30:44 WARN snappy.LoadSnappy: Snappy native library not loaded
> 13/07/11 15:30:44 INFO mapred.JobClient: Running job: job_201307111355_0015
> 13/07/11 15:30:45 INFO mapred.JobClient: map 0% reduce 0%
> 13/07/11 15:31:03 INFO mapred.JobClient: map 42% reduce 0%
> 13/07/11 15:31:06 INFO mapred.JobClient: map 57% reduce 0%
> 13/07/11 15:31:09 INFO mapred.JobClient: map 71% reduce 0%
> 13/07/11 15:31:15 INFO mapred.JobClient: map 100% reduce 0%
> 13/07/11 15:31:18 INFO mapred.JobClient: map 100% reduce 23%
> 13/07/11 15:31:27 INFO mapred.JobClient: map 100% reduce 100%
> 13/07/11 15:31:32 INFO mapred.JobClient: Job complete:
> job_201307111355_0015
> 13/07/11 15:31:32 INFO mapred.JobClient: Counters: 30
> 13/07/11 15:31:32 INFO mapred.JobClient: Job Counters
> 13/07/11 15:31:32 INFO mapred.JobClient: Launched reduce tasks=1
> 13/07/11 15:31:32 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=67576
> 13/07/11 15:31:32 INFO mapred.JobClient: Total time spent by all
> reduces waiting after reserving slots (ms)=0
> 13/07/11 15:31:32 INFO mapred.JobClient: Total time spent by all maps
> waiting after reserving slots (ms)=0
> 13/07/11 15:31:32 INFO mapred.JobClient: Rack-local map tasks=3
> 13/07/11 15:31:32 INFO mapred.JobClient: Launched map tasks=7
> 13/07/11 15:31:32 INFO mapred.JobClient: Data-local map tasks=4
> 13/07/11 15:31:32 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=21992
> 13/07/11 15:31:32 INFO mapred.JobClient: File Output Format Counters
> 13/07/11 15:31:32 INFO mapred.JobClient: Bytes Written=1412505
> 13/07/11 15:31:32 INFO mapred.JobClient: FileSystemCounters
> 13/07/11 15:31:32 INFO mapred.JobClient: FILE_BYTES_READ=5414195
> 13/07/11 15:31:32 INFO mapred.JobClient: HDFS_BYTES_READ=6950820
> 13/07/11 15:31:32 INFO mapred.JobClient: FILE_BYTES_WRITTEN=8744993
> 13/07/11 15:31:32 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=1412505
> 13/07/11 15:31:32 INFO mapred.JobClient: File Input Format Counters
> 13/07/11 15:31:32 INFO mapred.JobClient: Bytes Read=6950001
> 13/07/11 15:31:32 INFO mapred.JobClient: Map-Reduce Framework
> 13/07/11 15:31:32 INFO mapred.JobClient: Map output materialized
> bytes=3157469
> 13/07/11 15:31:32 INFO mapred.JobClient: Map input records=137146
> 13/07/11 15:31:32 INFO mapred.JobClient: Reduce shuffle bytes=2904836
> 13/07/11 15:31:32 INFO mapred.JobClient: Spilled Records=594764
> 13/07/11 15:31:32 INFO mapred.JobClient: Map output bytes=11435849
> 13/07/11 15:31:32 INFO mapred.JobClient: Total committed heap usage
> (bytes)=1128136704
> 13/07/11 15:31:32 INFO mapred.JobClient: CPU time spent (ms)=18230
> 13/07/11 15:31:32 INFO mapred.JobClient: Combine input records=1174991
> 13/07/11 15:31:32 INFO mapred.JobClient: SPLIT_RAW_BYTES=819
> 13/07/11 15:31:32 INFO mapred.JobClient: Reduce input records=218990
> 13/07/11 15:31:32 INFO mapred.JobClient: Reduce input groups=128513
> 13/07/11 15:31:32 INFO mapred.JobClient: Combine output records=218990
> 13/07/11 15:31:32 INFO mapred.JobClient: Physical memory (bytes)
> snapshot=1179656192
> 13/07/11 15:31:32 INFO mapred.JobClient: Reduce output records=128513
> 13/07/11 15:31:32 INFO mapred.JobClient: Virtual memory (bytes)
> snapshot=22992117760
> 13/07/11 15:31:32 INFO mapred.JobClient: Map output records=1174991
>
> from web interface (http://n1:50030/) I saw that both (n1 and n2 ) were
> used without any errors.
>
> Problems appear if I try to use following commands in master (n1):
>
> [hduser@n1 ~]$hadoop jar mahout-distribution-0.7/**mahout-examples-0.7-job.jar
> org.apache.mahout.classifier.**df.mapreduce.BuildForest
> -Dmapred.max.split.size=**1874231 -p -d testdata/bal_ee_2009.csv -ds
> testdata/bal_ee_2009.csv.info -sl 10 -o bal_ee_2009_out -t 1
>
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in [file:/usr/local/hadoop-1.0.4/**org/slf4j/impl/**
> StaticLoggerBinder.class]
> SLF4J: Found binding in [jar:file:/usr/local/hadoop-1.**
> 0.4/lib/slf4j-log4j12-1.4.3.**jar!/org/slf4j/impl/**
> StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.**html#multiple_bindings<http://www.slf4j.org/codes.html#multiple_bindings>for an explanation.
> 13/07/11 15:36:50 INFO mapreduce.BuildForest: Partial Mapred implementation
> 13/07/11 15:36:50 INFO mapreduce.BuildForest: Building the forest...
> 13/07/11 15:36:50 WARN mapred.JobClient: No job jar file set. User
> classes may not be found. See JobConf(Class) or JobConf#setJar(String).
> 13/07/11 15:36:50 INFO input.FileInputFormat: Total input paths to process
> : 1
> 13/07/11 15:36:50 INFO util.NativeCodeLoader: Loaded the native-hadoop
> library
> 13/07/11 15:36:50 WARN snappy.LoadSnappy: Snappy native library not loaded
> 13/07/11 15:36:50 INFO mapred.JobClient: Running job: job_201307111355_0016
> 13/07/11 15:36:51 INFO mapred.JobClient: map 0% reduce 0%
> 13/07/11 15:37:09 INFO mapred.JobClient: map 7% reduce 0%
> 13/07/11 15:37:11 INFO mapred.JobClient: Task Id :
> attempt_201307111355_0016_m_**000002_0, Status : FAILED
> java.lang.RuntimeException: java.lang.**ClassNotFoundException:
> org.apache.mahout.classifier.**df.mapreduce.partial.**Step1Mapper
> at org.apache.hadoop.conf.**Configuration.getClass(**
> Configuration.java:867)
> at org.apache.hadoop.mapreduce.**JobContext.getMapperClass(**
> JobContext.java:199)
> at org.apache.hadoop.mapred.**MapTask.runNewMapper(MapTask.**
> java:719)
> at org.apache.hadoop.mapred.**MapTask.run(MapTask.java:370)
> at org.apache.hadoop.mapred.**Child$4.run(Child.java:255)
> at java.security.**AccessController.doPrivileged(**Native Method)
> at javax.security.auth.Subject.**doAs(Subject.java:415)
> at org.apache.hadoop.security.**UserGroupInformation.doAs(**
> UserGroupInformation.java:**1121)
> at org.apache.hadoop.mapred.**Child.main(Child.java:249)
> Caused by: java.lang.**ClassNotFoundException:
> org.apache.mahout.classifier.**df.mapreduce.partial.**Step1Mapper
> at java.net.URLClassLoader$1.run(**URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(**URLClassLoader.java:355)
> at java.security.**AccessController.doPrivileged(**Native Method)
> at java.net.URLClassLoader.**findClass(URLClassLoader.java:**354)
> at java.lang.ClassLoader.**loadClass(ClassLoader.java:**424)
> at sun.misc.Launcher$**AppClassLoader.loadClass(**
> Launcher.java:308)
> at java.lang.ClassLoader.**loadClass(ClassLoader.java:**357)
> at java.lang.Class.forName0(**Native Method)
> at java.lang.Class.forName(Class.**java:270)
> at org.apache.hadoop.conf.**Configuration.getClassByName(**
> Configuration.java:820)
> at org.apache.hadoop.conf.**Configuration.getClass(**
> Configuration.java:865)
> ... 8 more
>
> ...
> ...
> ...
>
> 13/07/11 15:37:37 INFO mapred.JobClient: map 61% reduce 0%
> 13/07/11 15:37:46 INFO mapred.JobClient: map 76% reduce 0%
> 13/07/11 15:37:55 INFO mapred.JobClient: map 92% reduce 0%
> 13/07/11 15:38:01 INFO mapred.JobClient: map 100% reduce 0%
> 13/07/11 15:38:06 INFO mapred.JobClient: Job complete:
> job_201307111355_0016
> 13/07/11 15:38:06 INFO mapred.JobClient: Counters: 20
> 13/07/11 15:38:06 INFO mapred.JobClient: Job Counters
> 13/07/11 15:38:06 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=184724
> 13/07/11 15:38:06 INFO mapred.JobClient: Total time spent by all
> reduces waiting after reserving slots (ms)=0
> 13/07/11 15:38:06 INFO mapred.JobClient: Total time spent by all maps
> waiting after reserving slots (ms)=0
> 13/07/11 15:38:06 INFO mapred.JobClient: Rack-local map tasks=13
> 13/07/11 15:38:06 INFO mapred.JobClient: Launched map tasks=22
> 13/07/11 15:38:06 INFO mapred.JobClient: Data-local map tasks=9
> 13/07/11 15:38:06 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
> 13/07/11 15:38:06 INFO mapred.JobClient: File Output Format Counters
> 13/07/11 15:38:06 INFO mapred.JobClient: Bytes Written=184520
> 13/07/11 15:38:06 INFO mapred.JobClient: FileSystemCounters
> 13/07/11 15:38:06 INFO mapred.JobClient: HDFS_BYTES_READ=23444612
> 13/07/11 15:38:06 INFO mapred.JobClient: FILE_BYTES_WRITTEN=294037
> 13/07/11 15:38:06 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=184520
> 13/07/11 15:38:06 INFO mapred.JobClient: File Input Format Counters
> 13/07/11 15:38:06 INFO mapred.JobClient: Bytes Read=22952913
> 13/07/11 15:38:06 INFO mapred.JobClient: Map-Reduce Framework
> 13/07/11 15:38:06 INFO mapred.JobClient: Map input records=204091
> 13/07/11 15:38:06 INFO mapred.JobClient: Physical memory (bytes)
> snapshot=1402363904
> 13/07/11 15:38:06 INFO mapred.JobClient: Spilled Records=0
> 13/07/11 15:38:06 INFO mapred.JobClient: CPU time spent (ms)=26640
> 13/07/11 15:38:06 INFO mapred.JobClient: Total committed heap usage
> (bytes)=1333985280
> 13/07/11 15:38:06 INFO mapred.JobClient: Virtual memory (bytes)
> snapshot=37349126144
> 13/07/11 15:38:06 INFO mapred.JobClient: Map output records=1
> 13/07/11 15:38:06 INFO mapred.JobClient: SPLIT_RAW_BYTES=1573
> 13/07/11 15:38:07 INFO common.HadoopUtil: Deleting
> hdfs://master:54310/user/**hduser/bal_ee_2009_out
> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Build Time: 0h 1m 16s 977
> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Forest num Nodes: 10700
> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Forest mean num Nodes: 10700
> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Forest mean max Depth: 17
> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Storing the forest in:
> bal_ee_2009_out/forest.seq
>
> From web console I can see that n1 tasks are fine but n2 tasks failed:
>
> java.lang.RuntimeException: java.lang.**ClassNotFoundException:
> org.apache.mahout.classifier.**df.mapreduce.partial.**Step1Mapper
> at org.apache.hadoop.conf.**Configuration.getClass(**
> Configuration.java:867)
> at org.apache.hadoop.mapreduce.**JobContext.getMapperClass(**
> JobContext.java:199)
> at org.apache.hadoop.mapred.**MapTask.runNewMapper(MapTask.**
> java:719)
> at org.apache.hadoop.mapred.**MapTask.run(MapTask.java:370)
> at org.apache.hadoop.mapred.**Child$4.run(Child.java:255)
> at java.security.**AccessController.doPrivileged(**Native Method)
> at javax.security.auth.Subject.**doAs(Subject.java:415)
> at org.apache.hadoop.security.**UserGroupInformation.doAs(**
> UserGroupInformation.java:**1121)
> at org.apache.hadoop.mapred.**Child.main(Child.java:249)
> Caused by: java.lang.**ClassNotFoundException:
> org.apache.mahout.classifier.**df.mapreduce.partial.**Step1Mapper
> at java.net.URLClassLoader$1.run(**URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(**URLClassLoader.java:355)
> at java.security.**AccessController.doPrivileged(**Native Method)
> at java.net.URLClassLoader.**findClass(URLClassLoader.java:**354)
> at java.lang.ClassLoader.**loadClass(ClassLoader.java:**424)
> at sun.misc.Launcher$**AppClassLoader.loadClass(**
> Launcher.java:308)
> at java.lang.ClassLoader.**loadClass(ClassLoader.java:**357)
> at java.lang.Class.forName0(**Native Method)
> at java.lang.Class.forName(Class.**java:270)
> at org.apache.hadoop.conf.**Configuration.getClassByName(**
> Configuration.java:820)
> at org.apache.hadoop.conf.**Configuration.getClass(**
> Configuration.java:865)
> ... 8 more
>
>
> But If I run same command from n2
> [hduser@n2 ~]$ hadoop jar mahout-distribution-0.7/**mahout-examples-0.7-job.jar
> org.apache.mahout.classifier.**df.mapreduce.BuildForest
> -Dmapred.max.split.size=**1874231 -p -d testdata/bal_ee_2009.csv -ds
> testdata/bal_ee_2009.csv.info -sl 10 -o bal_ee_2009_out -t 1
>
> 13/07/11 15:41:20 INFO mapreduce.BuildForest: Partial Mapred implementation
> 13/07/11 15:41:20 INFO mapreduce.BuildForest: Building the forest...
> 13/07/11 15:41:21 INFO input.FileInputFormat: Total input paths to process
> : 1
> 13/07/11 15:41:21 INFO util.NativeCodeLoader: Loaded the native-hadoop
> library
> 13/07/11 15:41:21 WARN snappy.LoadSnappy: Snappy native library not loaded
> 13/07/11 15:41:21 INFO mapred.JobClient: Running job: job_201307111355_0017
> 13/07/11 15:41:22 INFO mapred.JobClient: map 0% reduce 0%
> 13/07/11 15:41:43 INFO mapred.JobClient: map 7% reduce 0%
> 13/07/11 15:41:46 INFO mapred.JobClient: map 15% reduce 0%
> 13/07/11 15:41:49 INFO mapred.JobClient: map 38% reduce 0%
> 13/07/11 15:41:52 INFO mapred.JobClient: map 46% reduce 0%
> 13/07/11 15:42:01 INFO mapred.JobClient: map 61% reduce 0%
> 13/07/11 15:42:02 INFO mapred.JobClient: map 76% reduce 0%
> 13/07/11 15:42:09 INFO mapred.JobClient: map 84% reduce 0%
> 13/07/11 15:42:13 INFO mapred.JobClient: map 100% reduce 0%
> 13/07/11 15:42:19 INFO mapred.JobClient: Job complete:
> job_201307111355_0017
> 13/07/11 15:42:19 INFO mapred.JobClient: Counters: 20
> 13/07/11 15:42:19 INFO mapred.JobClient: Job Counters
> 13/07/11 15:42:19 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=138911
> 13/07/11 15:42:19 INFO mapred.JobClient: Total time spent by all
> reduces waiting after reserving slots (ms)=0
> 13/07/11 15:42:19 INFO mapred.JobClient: Total time spent by all maps
> waiting after reserving slots (ms)=0
> 13/07/11 15:42:19 INFO mapred.JobClient: Rack-local map tasks=6
> 13/07/11 15:42:19 INFO mapred.JobClient: Launched map tasks=13
> 13/07/11 15:42:19 INFO mapred.JobClient: Data-local map tasks=7
> 13/07/11 15:42:19 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
> 13/07/11 15:42:19 INFO mapred.JobClient: File Output Format Counters
> 13/07/11 15:42:19 INFO mapred.JobClient: Bytes Written=184956
> 13/07/11 15:42:19 INFO mapred.JobClient: FileSystemCounters
> 13/07/11 15:42:19 INFO mapred.JobClient: HDFS_BYTES_READ=23444612
> 13/07/11 15:42:19 INFO mapred.JobClient: FILE_BYTES_WRITTEN=296026
> 13/07/11 15:42:19 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=184956
> 13/07/11 15:42:19 INFO mapred.JobClient: File Input Format Counters
> 13/07/11 15:42:19 INFO mapred.JobClient: Bytes Read=22952913
> 13/07/11 15:42:19 INFO mapred.JobClient: Map-Reduce Framework
> 13/07/11 15:42:19 INFO mapred.JobClient: Map input records=204091
> 13/07/11 15:42:19 INFO mapred.JobClient: Physical memory (bytes)
> snapshot=1370255360
> 13/07/11 15:42:19 INFO mapred.JobClient: Spilled Records=0
> 13/07/11 15:42:19 INFO mapred.JobClient: CPU time spent (ms)=27990
> 13/07/11 15:42:19 INFO mapred.JobClient: Total committed heap usage
> (bytes)=1292500992
> 13/07/11 15:42:19 INFO mapred.JobClient: Virtual memory (bytes)
> snapshot=37360082944
> 13/07/11 15:42:19 INFO mapred.JobClient: Map output records=1
> 13/07/11 15:42:19 INFO mapred.JobClient: SPLIT_RAW_BYTES=1573
> 13/07/11 15:42:20 INFO common.HadoopUtil: Deleting
> hdfs://master:54310/user/**hduser/bal_ee_2009_out
> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Build Time: 0h 0m 59s 386
> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Forest num Nodes: 10698
> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Forest mean num Nodes: 10698
> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Forest mean max Depth: 16
> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Storing the forest in:
> bal_ee_2009_out/forest.seq
>
> Both (n1 and n2) are used and from web console I can see that there are no
> errors.
>
> Is there any explanations why I am getting errors when I run command from
> master?
>
>
> --
> Regards, Margus (Margusja) Roo
> +372 51 48 780
> http://margus.roo.ee
> skype: margusja
> -----BEGIN PUBLIC KEY-----
> MIGfMA0GCSqGSIb3DQEBAQUAA4GNAD**CBiQKBgQCvbeg7LwEC2SCpAEewwpC3**ajxE
> 5ZsRMCB77L8bae9G7TslgLkoIzo9yO**jPdx2NN6DllKbV65UjTay43uUDyql9**g3tl
> RhiJIcoAExkSTykWqAIPR88LfilLy1**JlQ+0RD8OXiWOVVQfhOHpQ0R/**jcAkM2lZa
> BjM8j36yJvoBVsfOHQIDAQAB
> -----END PUBLIC KEY-----
>
>
Re: Task failure in slave node
Posted by Azuryy Yu <az...@gmail.com>.
hi,
put all mahou jars under hadoop_home/lib, then restart cluster.
On Jul 11, 2013 8:45 PM, "Margusja" <ma...@roo.ee> wrote:
> Hi
>
> I have tow nodes:
> n1 (master, salve) and n2 (slave)
>
> after set up I ran wordcount example and it worked fine:
> [hduser@n1 ~]$ hadoop jar /usr/local/hadoop/hadoop-**examples-1.0.4.jar
> wordcount /user/hduser/gutenberg /user/hduser/gutenberg-output
> 13/07/11 15:30:44 INFO input.FileInputFormat: Total input paths to process
> : 7
> 13/07/11 15:30:44 INFO util.NativeCodeLoader: Loaded the native-hadoop
> library
> 13/07/11 15:30:44 WARN snappy.LoadSnappy: Snappy native library not loaded
> 13/07/11 15:30:44 INFO mapred.JobClient: Running job: job_201307111355_0015
> 13/07/11 15:30:45 INFO mapred.JobClient: map 0% reduce 0%
> 13/07/11 15:31:03 INFO mapred.JobClient: map 42% reduce 0%
> 13/07/11 15:31:06 INFO mapred.JobClient: map 57% reduce 0%
> 13/07/11 15:31:09 INFO mapred.JobClient: map 71% reduce 0%
> 13/07/11 15:31:15 INFO mapred.JobClient: map 100% reduce 0%
> 13/07/11 15:31:18 INFO mapred.JobClient: map 100% reduce 23%
> 13/07/11 15:31:27 INFO mapred.JobClient: map 100% reduce 100%
> 13/07/11 15:31:32 INFO mapred.JobClient: Job complete:
> job_201307111355_0015
> 13/07/11 15:31:32 INFO mapred.JobClient: Counters: 30
> 13/07/11 15:31:32 INFO mapred.JobClient: Job Counters
> 13/07/11 15:31:32 INFO mapred.JobClient: Launched reduce tasks=1
> 13/07/11 15:31:32 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=67576
> 13/07/11 15:31:32 INFO mapred.JobClient: Total time spent by all
> reduces waiting after reserving slots (ms)=0
> 13/07/11 15:31:32 INFO mapred.JobClient: Total time spent by all maps
> waiting after reserving slots (ms)=0
> 13/07/11 15:31:32 INFO mapred.JobClient: Rack-local map tasks=3
> 13/07/11 15:31:32 INFO mapred.JobClient: Launched map tasks=7
> 13/07/11 15:31:32 INFO mapred.JobClient: Data-local map tasks=4
> 13/07/11 15:31:32 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=21992
> 13/07/11 15:31:32 INFO mapred.JobClient: File Output Format Counters
> 13/07/11 15:31:32 INFO mapred.JobClient: Bytes Written=1412505
> 13/07/11 15:31:32 INFO mapred.JobClient: FileSystemCounters
> 13/07/11 15:31:32 INFO mapred.JobClient: FILE_BYTES_READ=5414195
> 13/07/11 15:31:32 INFO mapred.JobClient: HDFS_BYTES_READ=6950820
> 13/07/11 15:31:32 INFO mapred.JobClient: FILE_BYTES_WRITTEN=8744993
> 13/07/11 15:31:32 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=1412505
> 13/07/11 15:31:32 INFO mapred.JobClient: File Input Format Counters
> 13/07/11 15:31:32 INFO mapred.JobClient: Bytes Read=6950001
> 13/07/11 15:31:32 INFO mapred.JobClient: Map-Reduce Framework
> 13/07/11 15:31:32 INFO mapred.JobClient: Map output materialized
> bytes=3157469
> 13/07/11 15:31:32 INFO mapred.JobClient: Map input records=137146
> 13/07/11 15:31:32 INFO mapred.JobClient: Reduce shuffle bytes=2904836
> 13/07/11 15:31:32 INFO mapred.JobClient: Spilled Records=594764
> 13/07/11 15:31:32 INFO mapred.JobClient: Map output bytes=11435849
> 13/07/11 15:31:32 INFO mapred.JobClient: Total committed heap usage
> (bytes)=1128136704
> 13/07/11 15:31:32 INFO mapred.JobClient: CPU time spent (ms)=18230
> 13/07/11 15:31:32 INFO mapred.JobClient: Combine input records=1174991
> 13/07/11 15:31:32 INFO mapred.JobClient: SPLIT_RAW_BYTES=819
> 13/07/11 15:31:32 INFO mapred.JobClient: Reduce input records=218990
> 13/07/11 15:31:32 INFO mapred.JobClient: Reduce input groups=128513
> 13/07/11 15:31:32 INFO mapred.JobClient: Combine output records=218990
> 13/07/11 15:31:32 INFO mapred.JobClient: Physical memory (bytes)
> snapshot=1179656192
> 13/07/11 15:31:32 INFO mapred.JobClient: Reduce output records=128513
> 13/07/11 15:31:32 INFO mapred.JobClient: Virtual memory (bytes)
> snapshot=22992117760
> 13/07/11 15:31:32 INFO mapred.JobClient: Map output records=1174991
>
> from web interface (http://n1:50030/) I saw that both (n1 and n2 ) were
> used without any errors.
>
> Problems appear if I try to use following commands in master (n1):
>
> [hduser@n1 ~]$hadoop jar mahout-distribution-0.7/**mahout-examples-0.7-job.jar
> org.apache.mahout.classifier.**df.mapreduce.BuildForest
> -Dmapred.max.split.size=**1874231 -p -d testdata/bal_ee_2009.csv -ds
> testdata/bal_ee_2009.csv.info -sl 10 -o bal_ee_2009_out -t 1
>
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in [file:/usr/local/hadoop-1.0.4/**org/slf4j/impl/**
> StaticLoggerBinder.class]
> SLF4J: Found binding in [jar:file:/usr/local/hadoop-1.**
> 0.4/lib/slf4j-log4j12-1.4.3.**jar!/org/slf4j/impl/**
> StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.**html#multiple_bindings<http://www.slf4j.org/codes.html#multiple_bindings>for an explanation.
> 13/07/11 15:36:50 INFO mapreduce.BuildForest: Partial Mapred implementation
> 13/07/11 15:36:50 INFO mapreduce.BuildForest: Building the forest...
> 13/07/11 15:36:50 WARN mapred.JobClient: No job jar file set. User
> classes may not be found. See JobConf(Class) or JobConf#setJar(String).
> 13/07/11 15:36:50 INFO input.FileInputFormat: Total input paths to process
> : 1
> 13/07/11 15:36:50 INFO util.NativeCodeLoader: Loaded the native-hadoop
> library
> 13/07/11 15:36:50 WARN snappy.LoadSnappy: Snappy native library not loaded
> 13/07/11 15:36:50 INFO mapred.JobClient: Running job: job_201307111355_0016
> 13/07/11 15:36:51 INFO mapred.JobClient: map 0% reduce 0%
> 13/07/11 15:37:09 INFO mapred.JobClient: map 7% reduce 0%
> 13/07/11 15:37:11 INFO mapred.JobClient: Task Id :
> attempt_201307111355_0016_m_**000002_0, Status : FAILED
> java.lang.RuntimeException: java.lang.**ClassNotFoundException:
> org.apache.mahout.classifier.**df.mapreduce.partial.**Step1Mapper
> at org.apache.hadoop.conf.**Configuration.getClass(**
> Configuration.java:867)
> at org.apache.hadoop.mapreduce.**JobContext.getMapperClass(**
> JobContext.java:199)
> at org.apache.hadoop.mapred.**MapTask.runNewMapper(MapTask.**
> java:719)
> at org.apache.hadoop.mapred.**MapTask.run(MapTask.java:370)
> at org.apache.hadoop.mapred.**Child$4.run(Child.java:255)
> at java.security.**AccessController.doPrivileged(**Native Method)
> at javax.security.auth.Subject.**doAs(Subject.java:415)
> at org.apache.hadoop.security.**UserGroupInformation.doAs(**
> UserGroupInformation.java:**1121)
> at org.apache.hadoop.mapred.**Child.main(Child.java:249)
> Caused by: java.lang.**ClassNotFoundException:
> org.apache.mahout.classifier.**df.mapreduce.partial.**Step1Mapper
> at java.net.URLClassLoader$1.run(**URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(**URLClassLoader.java:355)
> at java.security.**AccessController.doPrivileged(**Native Method)
> at java.net.URLClassLoader.**findClass(URLClassLoader.java:**354)
> at java.lang.ClassLoader.**loadClass(ClassLoader.java:**424)
> at sun.misc.Launcher$**AppClassLoader.loadClass(**
> Launcher.java:308)
> at java.lang.ClassLoader.**loadClass(ClassLoader.java:**357)
> at java.lang.Class.forName0(**Native Method)
> at java.lang.Class.forName(Class.**java:270)
> at org.apache.hadoop.conf.**Configuration.getClassByName(**
> Configuration.java:820)
> at org.apache.hadoop.conf.**Configuration.getClass(**
> Configuration.java:865)
> ... 8 more
>
> ...
> ...
> ...
>
> 13/07/11 15:37:37 INFO mapred.JobClient: map 61% reduce 0%
> 13/07/11 15:37:46 INFO mapred.JobClient: map 76% reduce 0%
> 13/07/11 15:37:55 INFO mapred.JobClient: map 92% reduce 0%
> 13/07/11 15:38:01 INFO mapred.JobClient: map 100% reduce 0%
> 13/07/11 15:38:06 INFO mapred.JobClient: Job complete:
> job_201307111355_0016
> 13/07/11 15:38:06 INFO mapred.JobClient: Counters: 20
> 13/07/11 15:38:06 INFO mapred.JobClient: Job Counters
> 13/07/11 15:38:06 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=184724
> 13/07/11 15:38:06 INFO mapred.JobClient: Total time spent by all
> reduces waiting after reserving slots (ms)=0
> 13/07/11 15:38:06 INFO mapred.JobClient: Total time spent by all maps
> waiting after reserving slots (ms)=0
> 13/07/11 15:38:06 INFO mapred.JobClient: Rack-local map tasks=13
> 13/07/11 15:38:06 INFO mapred.JobClient: Launched map tasks=22
> 13/07/11 15:38:06 INFO mapred.JobClient: Data-local map tasks=9
> 13/07/11 15:38:06 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
> 13/07/11 15:38:06 INFO mapred.JobClient: File Output Format Counters
> 13/07/11 15:38:06 INFO mapred.JobClient: Bytes Written=184520
> 13/07/11 15:38:06 INFO mapred.JobClient: FileSystemCounters
> 13/07/11 15:38:06 INFO mapred.JobClient: HDFS_BYTES_READ=23444612
> 13/07/11 15:38:06 INFO mapred.JobClient: FILE_BYTES_WRITTEN=294037
> 13/07/11 15:38:06 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=184520
> 13/07/11 15:38:06 INFO mapred.JobClient: File Input Format Counters
> 13/07/11 15:38:06 INFO mapred.JobClient: Bytes Read=22952913
> 13/07/11 15:38:06 INFO mapred.JobClient: Map-Reduce Framework
> 13/07/11 15:38:06 INFO mapred.JobClient: Map input records=204091
> 13/07/11 15:38:06 INFO mapred.JobClient: Physical memory (bytes)
> snapshot=1402363904
> 13/07/11 15:38:06 INFO mapred.JobClient: Spilled Records=0
> 13/07/11 15:38:06 INFO mapred.JobClient: CPU time spent (ms)=26640
> 13/07/11 15:38:06 INFO mapred.JobClient: Total committed heap usage
> (bytes)=1333985280
> 13/07/11 15:38:06 INFO mapred.JobClient: Virtual memory (bytes)
> snapshot=37349126144
> 13/07/11 15:38:06 INFO mapred.JobClient: Map output records=1
> 13/07/11 15:38:06 INFO mapred.JobClient: SPLIT_RAW_BYTES=1573
> 13/07/11 15:38:07 INFO common.HadoopUtil: Deleting
> hdfs://master:54310/user/**hduser/bal_ee_2009_out
> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Build Time: 0h 1m 16s 977
> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Forest num Nodes: 10700
> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Forest mean num Nodes: 10700
> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Forest mean max Depth: 17
> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Storing the forest in:
> bal_ee_2009_out/forest.seq
>
> From web console I can see that n1 tasks are fine but n2 tasks failed:
>
> java.lang.RuntimeException: java.lang.**ClassNotFoundException:
> org.apache.mahout.classifier.**df.mapreduce.partial.**Step1Mapper
> at org.apache.hadoop.conf.**Configuration.getClass(**
> Configuration.java:867)
> at org.apache.hadoop.mapreduce.**JobContext.getMapperClass(**
> JobContext.java:199)
> at org.apache.hadoop.mapred.**MapTask.runNewMapper(MapTask.**
> java:719)
> at org.apache.hadoop.mapred.**MapTask.run(MapTask.java:370)
> at org.apache.hadoop.mapred.**Child$4.run(Child.java:255)
> at java.security.**AccessController.doPrivileged(**Native Method)
> at javax.security.auth.Subject.**doAs(Subject.java:415)
> at org.apache.hadoop.security.**UserGroupInformation.doAs(**
> UserGroupInformation.java:**1121)
> at org.apache.hadoop.mapred.**Child.main(Child.java:249)
> Caused by: java.lang.**ClassNotFoundException:
> org.apache.mahout.classifier.**df.mapreduce.partial.**Step1Mapper
> at java.net.URLClassLoader$1.run(**URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(**URLClassLoader.java:355)
> at java.security.**AccessController.doPrivileged(**Native Method)
> at java.net.URLClassLoader.**findClass(URLClassLoader.java:**354)
> at java.lang.ClassLoader.**loadClass(ClassLoader.java:**424)
> at sun.misc.Launcher$**AppClassLoader.loadClass(**
> Launcher.java:308)
> at java.lang.ClassLoader.**loadClass(ClassLoader.java:**357)
> at java.lang.Class.forName0(**Native Method)
> at java.lang.Class.forName(Class.**java:270)
> at org.apache.hadoop.conf.**Configuration.getClassByName(**
> Configuration.java:820)
> at org.apache.hadoop.conf.**Configuration.getClass(**
> Configuration.java:865)
> ... 8 more
>
>
> But If I run same command from n2
> [hduser@n2 ~]$ hadoop jar mahout-distribution-0.7/**mahout-examples-0.7-job.jar
> org.apache.mahout.classifier.**df.mapreduce.BuildForest
> -Dmapred.max.split.size=**1874231 -p -d testdata/bal_ee_2009.csv -ds
> testdata/bal_ee_2009.csv.info -sl 10 -o bal_ee_2009_out -t 1
>
> 13/07/11 15:41:20 INFO mapreduce.BuildForest: Partial Mapred implementation
> 13/07/11 15:41:20 INFO mapreduce.BuildForest: Building the forest...
> 13/07/11 15:41:21 INFO input.FileInputFormat: Total input paths to process
> : 1
> 13/07/11 15:41:21 INFO util.NativeCodeLoader: Loaded the native-hadoop
> library
> 13/07/11 15:41:21 WARN snappy.LoadSnappy: Snappy native library not loaded
> 13/07/11 15:41:21 INFO mapred.JobClient: Running job: job_201307111355_0017
> 13/07/11 15:41:22 INFO mapred.JobClient: map 0% reduce 0%
> 13/07/11 15:41:43 INFO mapred.JobClient: map 7% reduce 0%
> 13/07/11 15:41:46 INFO mapred.JobClient: map 15% reduce 0%
> 13/07/11 15:41:49 INFO mapred.JobClient: map 38% reduce 0%
> 13/07/11 15:41:52 INFO mapred.JobClient: map 46% reduce 0%
> 13/07/11 15:42:01 INFO mapred.JobClient: map 61% reduce 0%
> 13/07/11 15:42:02 INFO mapred.JobClient: map 76% reduce 0%
> 13/07/11 15:42:09 INFO mapred.JobClient: map 84% reduce 0%
> 13/07/11 15:42:13 INFO mapred.JobClient: map 100% reduce 0%
> 13/07/11 15:42:19 INFO mapred.JobClient: Job complete:
> job_201307111355_0017
> 13/07/11 15:42:19 INFO mapred.JobClient: Counters: 20
> 13/07/11 15:42:19 INFO mapred.JobClient: Job Counters
> 13/07/11 15:42:19 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=138911
> 13/07/11 15:42:19 INFO mapred.JobClient: Total time spent by all
> reduces waiting after reserving slots (ms)=0
> 13/07/11 15:42:19 INFO mapred.JobClient: Total time spent by all maps
> waiting after reserving slots (ms)=0
> 13/07/11 15:42:19 INFO mapred.JobClient: Rack-local map tasks=6
> 13/07/11 15:42:19 INFO mapred.JobClient: Launched map tasks=13
> 13/07/11 15:42:19 INFO mapred.JobClient: Data-local map tasks=7
> 13/07/11 15:42:19 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
> 13/07/11 15:42:19 INFO mapred.JobClient: File Output Format Counters
> 13/07/11 15:42:19 INFO mapred.JobClient: Bytes Written=184956
> 13/07/11 15:42:19 INFO mapred.JobClient: FileSystemCounters
> 13/07/11 15:42:19 INFO mapred.JobClient: HDFS_BYTES_READ=23444612
> 13/07/11 15:42:19 INFO mapred.JobClient: FILE_BYTES_WRITTEN=296026
> 13/07/11 15:42:19 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=184956
> 13/07/11 15:42:19 INFO mapred.JobClient: File Input Format Counters
> 13/07/11 15:42:19 INFO mapred.JobClient: Bytes Read=22952913
> 13/07/11 15:42:19 INFO mapred.JobClient: Map-Reduce Framework
> 13/07/11 15:42:19 INFO mapred.JobClient: Map input records=204091
> 13/07/11 15:42:19 INFO mapred.JobClient: Physical memory (bytes)
> snapshot=1370255360
> 13/07/11 15:42:19 INFO mapred.JobClient: Spilled Records=0
> 13/07/11 15:42:19 INFO mapred.JobClient: CPU time spent (ms)=27990
> 13/07/11 15:42:19 INFO mapred.JobClient: Total committed heap usage
> (bytes)=1292500992
> 13/07/11 15:42:19 INFO mapred.JobClient: Virtual memory (bytes)
> snapshot=37360082944
> 13/07/11 15:42:19 INFO mapred.JobClient: Map output records=1
> 13/07/11 15:42:19 INFO mapred.JobClient: SPLIT_RAW_BYTES=1573
> 13/07/11 15:42:20 INFO common.HadoopUtil: Deleting
> hdfs://master:54310/user/**hduser/bal_ee_2009_out
> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Build Time: 0h 0m 59s 386
> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Forest num Nodes: 10698
> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Forest mean num Nodes: 10698
> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Forest mean max Depth: 16
> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Storing the forest in:
> bal_ee_2009_out/forest.seq
>
> Both (n1 and n2) are used and from web console I can see that there are no
> errors.
>
> Is there any explanations why I am getting errors when I run command from
> master?
>
>
> --
> Regards, Margus (Margusja) Roo
> +372 51 48 780
> http://margus.roo.ee
> skype: margusja
> -----BEGIN PUBLIC KEY-----
> MIGfMA0GCSqGSIb3DQEBAQUAA4GNAD**CBiQKBgQCvbeg7LwEC2SCpAEewwpC3**ajxE
> 5ZsRMCB77L8bae9G7TslgLkoIzo9yO**jPdx2NN6DllKbV65UjTay43uUDyql9**g3tl
> RhiJIcoAExkSTykWqAIPR88LfilLy1**JlQ+0RD8OXiWOVVQfhOHpQ0R/**jcAkM2lZa
> BjM8j36yJvoBVsfOHQIDAQAB
> -----END PUBLIC KEY-----
>
>
Re: Task failure in slave node
Posted by "devaraj.k@huawei.com" <de...@huawei.com>.
Hi,
It seems mahout-examples-0.7-job.jar is depending on other jars/classes.
While running Job Tasks it is not able to find those classes in the
classpath and failing those tasks.
You need to provide the dependent jar files while submitting/running Job.
Thanks
Devaraj k
--
View this message in context: http://lucene.472066.n3.nabble.com/Task-failure-in-slave-node-tp4077284p4077290.html
Sent from the Hadoop lucene-users mailing list archive at Nabble.com.