You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by Mishari Almishari <ma...@gmail.com> on 2010/07/12 20:24:37 UTC
question about the Twenty Newsgroup example
Hi,
I am new to Mahout and i am facing the following problem in the Twenty
Newsgroup example:
When I try to run the test command over hadoop
HADOOP_HOME/bin/hadoop \
jar \
$MAHOUT_HOME/examples/ .....
I got the following error: "java.lang.OutOfMemoryError: Java heap space"
you know how i fix the problem?
Re: question about the Twenty Newsgroup example
Posted by Mishari Almishari <ma...@gmail.com>.
Hi, Thanks for your reply!
I tried the fix you mentioned but i got the following error:
java.lang.RuntimeException: Error in configuring object
Any clue?
Here kind of the error in more details....
java.lang.RuntimeException: Error in configuring object
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:354)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
... 5 more
Caused by: java.lang.RuntimeException: Error in configuring object
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
... 10 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
... 13 more
Caused by: java.lang.OutOfMemoryError: Java heap space
at
org.apache.mahout.math.map.OpenIntObjectHashMap.rehash(OpenIntObjectHashMap.java:419)
at
org.apache.mahout.math.map.OpenIntObjectHashMap.put(OpenIntObjectHashMap.java:384)
at org.apache.mahout.math.SparseMatrix.setQuick(SparseMatrix.java:96)
at
org.apache.mahout.classifier.bayes.datastore.InMemoryBayesDatastore.loadFeatureWeight(InMemoryBayesDatastore.java:161)
at
org.apache.mahout.classifier.bayes.io.SequenceFileModelReader.loadWeightMatrix(SequenceFileModelReader.java:77)
at
org.apache.mahout.classifier.bayes.io.SequenceFileModelReader.loadModel(SequenceFileModelReader.java:56)
at
org.apache.mahout.classifier.bayes.datastore.InMemoryBayesDatastore.initialize(InMemoryBayesDatastore.java:80)
at
org.apache.mahout.classifier.bayes.model.ClassifierContext.initialize(ClassifierContext.java:47)
at
org.apache.mahout.classifier.bayes.mapreduce.bayes.BayesClassifierMapper.configure(BayesClassifierMapper.java:135)
... 18 more
10/07/12 15:48:44 INFO mapred.JobClient: Job complete: job_local_0001
10/07/12 15:48:44 INFO mapred.JobClient: Counters: 9
10/07/12 15:48:44 INFO mapred.JobClient: FileSystemCounters
10/07/12 15:48:44 INFO mapred.JobClient: FILE_BYTES_READ=81187185
10/07/12 15:48:44 INFO mapred.JobClient: FILE_BYTES_WRITTEN=14564458
10/07/12 15:48:44 INFO mapred.JobClient: Map-Reduce Framework
10/07/12 15:48:44 INFO mapred.JobClient: Combine output records=0
10/07/12 15:48:44 INFO mapred.JobClient: Map input records=0
10/07/12 15:48:44 INFO mapred.JobClient: Spilled Records=0
10/07/12 15:48:44 INFO mapred.JobClient: Map output bytes=0
10/07/12 15:48:44 INFO mapred.JobClient: Map input bytes=0
10/07/12 15:48:44 INFO mapred.JobClient: Map output records=0
10/07/12 15:48:44 INFO mapred.JobClient: Combine input records=0
Exception in thread "main" java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
at
org.apache.mahout.classifier.bayes.mapreduce.bayes.BayesClassifierDriver.runJob(BayesClassifierDriver.java:83)
at
org.apache.mahout.classifier.bayes.TestClassifier.classifyParallel(TestClassifier.java:288)
at
org.apache.mahout.classifier.bayes.TestClassifier.main(TestClassifier.java:178)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
On Mon, Jul 12, 2010 at 3:43 PM, Sean Owen <sr...@gmail.com> wrote:
> OutOfMemoryError from workers, right?
>
> You probably need to give more memory to the Hadoop workers. I do this
> by setting something like this in mapred-site.xml (there are other
> similar ways of doing this):
>
> <property>
> <name>mapred.child.java.opts</name>
> <value>-Xmx256m</value>
> </property>
>
> This gives each a 256MB heap. I don't know how much is required for 20
> news groups but this is the likely setting you need.
>
> On Mon, Jul 12, 2010 at 7:24 PM, Mishari Almishari <ma...@gmail.com>
> wrote:
> > Hi,
> > I am new to Mahout and i am facing the following problem in the Twenty
> > Newsgroup example:
> > When I try to run the test command over hadoop
> >
> > HADOOP_HOME/bin/hadoop \
> > jar \
> > $MAHOUT_HOME/examples/ .....
> >
> >
> > I got the following error: "java.lang.OutOfMemoryError: Java heap space"
> >
> > you know how i fix the problem?
> >
>
Re: question about the Twenty Newsgroup example
Posted by Sean Owen <sr...@gmail.com>.
OutOfMemoryError from workers, right?
You probably need to give more memory to the Hadoop workers. I do this
by setting something like this in mapred-site.xml (there are other
similar ways of doing this):
<property>
<name>mapred.child.java.opts</name>
<value>-Xmx256m</value>
</property>
This gives each a 256MB heap. I don't know how much is required for 20
news groups but this is the likely setting you need.
On Mon, Jul 12, 2010 at 7:24 PM, Mishari Almishari <ma...@gmail.com> wrote:
> Hi,
> I am new to Mahout and i am facing the following problem in the Twenty
> Newsgroup example:
> When I try to run the test command over hadoop
>
> HADOOP_HOME/bin/hadoop \
> jar \
> $MAHOUT_HOME/examples/ .....
>
>
> I got the following error: "java.lang.OutOfMemoryError: Java heap space"
>
> you know how i fix the problem?
>