You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by deneche abdelhakim <a_...@yahoo.fr> on 2008/06/02 11:50:20 UTC
OutOfMemory Exception !
I checked the last version of Mahout (rev. 662372) and got the following exception with many tests (the list of this tests is at the end of this post):
java.io.IOException: Job failed!
the following message is printed in System.err :
java.lang.OutOfMemoryError: Java heap space
I think its somehow caused by using Hadoop 0.17.0 as my own tests run perfectly with Hadoop 0.16.4
here are the tests that don't pass
org.apache.mahout.clustering.canopy.TestCanopyCreation:
testCanopyGenManhattanMR
testCanopyGenEuclideanMR
testClusteringManhattanMR
testClusteringEuclideanMR
testClusteringManhattanMRWithPayload
testClusteringEuclideanMRWithPayload
testUserDefinedDistanceMeasure
org.apache.mahout.clustering.meanshift.TestMeanShift
testCanopyEuclideanMRJob
__________________________________________________
Do You Yahoo!?
En finir avec le spam? Yahoo! Mail vous offre la meilleure protection possible contre les messages non sollicités
http://mail.yahoo.fr Yahoo! Mail
Re: OutOfMemory Exception !
Posted by Karl Wettin <ka...@gmail.com>.
2 jun 2008 kl. 13.11 skrev deneche abdelhakim:
>
> what is the minimal physical memory size needed to run Mahout ?
There has been efforts put in figuring that out and I suspect it will
differ quite a bit depending what algorithm you are using and on what
data set. Running tests might actually consume more resources than
using them for real, but I'm not sure.
Do you have the same problems if you run the tests one by one and not
the whole suite?
karl
Re: OutOfMemory Exception !
Posted by deneche abdelhakim <a_...@yahoo.fr>.
> De: Karl Wettin <ka...@gmail.com>
> Date: Lundi 2 Juin 2008, 12h24
>
> Do you run the tests from Ant or your development
> environment? If the
> latter, did you set vm arg Xmx to something?
they don't pass from Ant and from Eclipse, and no I didn't set Xmx (although I tried it and nothing changed)
I think the error is caused by my low physical memory size (512 mb), I debugged the code and found that by setting the following job parameter
io.sort.mb=10
makes my (own) tests run without problem
what is the minimal physical memory size needed to run Mahout ?
>
>
> 2 jun 2008 kl. 11.50 skrev deneche abdelhakim:
> > I checked the last version of Mahout (rev. 662372) and
> got the
> > following exception with many tests (the list of this
> tests is at
> > the end of this post):
> >
> > java.io.IOException: Job failed!
> >
> > the following message is printed in System.err :
> >
> > java.lang.OutOfMemoryError: Java heap space
> >
> > I think its somehow caused by using Hadoop 0.17.0 as
> my own tests
> > run perfectly with Hadoop 0.16.4
> >
> > here are the tests that don't pass
> >
> >
> org.apache.mahout.clustering.canopy.TestCanopyCreation:
> > testCanopyGenManhattanMR
> > testCanopyGenEuclideanMR
> > testClusteringManhattanMR
> > testClusteringEuclideanMR
> > testClusteringManhattanMRWithPayload
> > testClusteringEuclideanMRWithPayload
> > testUserDefinedDistanceMeasure
> >
> > org.apache.mahout.clustering.meanshift.TestMeanShift
> > testCanopyEuclideanMRJob
> >
> > __________________________________________________
> > Do You Yahoo!?
> > En finir avec le spam? Yahoo! Mail vous offre la
> meilleure
> > protection possible contre les messages non
> sollicités
> > http://mail.yahoo.fr Yahoo! Mail
__________________________________________________
Do You Yahoo!?
En finir avec le spam? Yahoo! Mail vous offre la meilleure protection possible contre les messages non sollicités
http://mail.yahoo.fr Yahoo! Mail
Re: OutOfMemory Exception !
Posted by Karl Wettin <ka...@gmail.com>.
Do you run the tests from Ant or your development environment? If the
latter, did you set vm arg Xmx to something?
karl
2 jun 2008 kl. 11.50 skrev deneche abdelhakim:
> I checked the last version of Mahout (rev. 662372) and got the
> following exception with many tests (the list of this tests is at
> the end of this post):
>
> java.io.IOException: Job failed!
>
> the following message is printed in System.err :
>
> java.lang.OutOfMemoryError: Java heap space
>
> I think its somehow caused by using Hadoop 0.17.0 as my own tests
> run perfectly with Hadoop 0.16.4
>
> here are the tests that don't pass
>
> org.apache.mahout.clustering.canopy.TestCanopyCreation:
> testCanopyGenManhattanMR
> testCanopyGenEuclideanMR
> testClusteringManhattanMR
> testClusteringEuclideanMR
> testClusteringManhattanMRWithPayload
> testClusteringEuclideanMRWithPayload
> testUserDefinedDistanceMeasure
>
> org.apache.mahout.clustering.meanshift.TestMeanShift
> testCanopyEuclideanMRJob
>
> __________________________________________________
> Do You Yahoo!?
> En finir avec le spam? Yahoo! Mail vous offre la meilleure
> protection possible contre les messages non sollicités
> http://mail.yahoo.fr Yahoo! Mail
Re: OutOfMemory Exception !
Posted by deneche abdelhakim <a_...@yahoo.fr>.
thanks it just worked fine :)
--- En date de : Lun 2.6.08, Sean Owen <sr...@gmail.com> a écrit :
> De: Sean Owen <sr...@gmail.com>
> Objet: Re: OutOfMemory Exception !
> À: mahout-dev@lucene.apache.org
> Date: Lundi 2 Juin 2008, 15h18
> Yep I saw the same thing. You can either change that
> settings up your
> maximum heap size to, say, 128MB. (Add -Xmx128m to the
> command line.)
> I just committed a change to build.xml to address this.
> Otherwise the
> JVM default is something like 64MB. It's not that your
> computer is
> out of memory.
>
> Yes it just looks like the new default settings assume more
> memory. It
> can be tuned but it seems like 128MB seems like an OK
> amount of memory
> for a worker to use.
>
> Sean
>
> On Mon, Jun 2, 2008 at 8:55 AM, Grant Ingersoll
> <gs...@apache.org> wrote:
> > I'm getting the same errors upon upgrading. We
> should look at the Hadoop 17
> > changes.
> >
> > On Jun 2, 2008, at 5:50 AM, deneche abdelhakim wrote:
> >
> >> I checked the last version of Mahout (rev. 662372)
> and got the following
> >> exception with many tests (the list of this tests
> is at the end of this
> >> post):
> >>
> >> java.io.IOException: Job failed!
> >>
> >> the following message is printed in System.err :
> >>
> >> java.lang.OutOfMemoryError: Java heap space
> >>
> >> I think its somehow caused by using Hadoop 0.17.0
> as my own tests run
> >> perfectly with Hadoop 0.16.4
> >>
> >> here are the tests that don't pass
> >>
> >>
> org.apache.mahout.clustering.canopy.TestCanopyCreation:
> >> testCanopyGenManhattanMR
> >> testCanopyGenEuclideanMR
> >> testClusteringManhattanMR
> >> testClusteringEuclideanMR
> >> testClusteringManhattanMRWithPayload
> >> testClusteringEuclideanMRWithPayload
> >> testUserDefinedDistanceMeasure
> >>
> >>
> org.apache.mahout.clustering.meanshift.TestMeanShift
> >> testCanopyEuclideanMRJob
> >>
> >> __________________________________________________
> >> Do You Yahoo!?
> >> En finir avec le spam? Yahoo! Mail vous offre la
> meilleure protection
> >> possible contre les messages non sollicités
> >> http://mail.yahoo.fr Yahoo! Mail
> >
> > --------------------------
> > Grant Ingersoll
> > http://www.lucidimagination.com
> >
> >
> >
__________________________________________________
Do You Yahoo!?
En finir avec le spam? Yahoo! Mail vous offre la meilleure protection possible contre les messages non sollicités
http://mail.yahoo.fr Yahoo! Mail
Re: OutOfMemory Exception !
Posted by Sean Owen <sr...@gmail.com>.
I just increased the max to 256M.
On Thu, Aug 7, 2008 at 3:22 PM, Allen Day <al...@gmail.com> wrote:
> Hey,
>
> This is still too low. 256m is sufficient for my tests to pass.
> I was getting failures with jdk 1.6.0_10-beta on a 64bit system.
>
> -Allen
Re: OutOfMemory Exception !
Posted by Allen Day <al...@gmail.com>.
Hey,
This is still too low. 256m is sufficient for my tests to pass.
I was getting failures with jdk 1.6.0_10-beta on a 64bit system.
-Allen
On Mon, Jun 2, 2008 at 6:18 AM, Sean Owen <sr...@gmail.com> wrote:
> Yep I saw the same thing. You can either change that settings up your
> maximum heap size to, say, 128MB. (Add -Xmx128m to the command line.)
> I just committed a change to build.xml to address this. Otherwise the
> JVM default is something like 64MB. It's not that your computer is
> out of memory.
>
> Yes it just looks like the new default settings assume more memory. It
> can be tuned but it seems like 128MB seems like an OK amount of memory
> for a worker to use.
>
> Sean
>
> On Mon, Jun 2, 2008 at 8:55 AM, Grant Ingersoll <gs...@apache.org> wrote:
>> I'm getting the same errors upon upgrading. We should look at the Hadoop 17
>> changes.
>>
>> On Jun 2, 2008, at 5:50 AM, deneche abdelhakim wrote:
>>
>>> I checked the last version of Mahout (rev. 662372) and got the following
>>> exception with many tests (the list of this tests is at the end of this
>>> post):
>>>
>>> java.io.IOException: Job failed!
>>>
>>> the following message is printed in System.err :
>>>
>>> java.lang.OutOfMemoryError: Java heap space
>>>
>>> I think its somehow caused by using Hadoop 0.17.0 as my own tests run
>>> perfectly with Hadoop 0.16.4
>>>
>>> here are the tests that don't pass
>>>
>>> org.apache.mahout.clustering.canopy.TestCanopyCreation:
>>> testCanopyGenManhattanMR
>>> testCanopyGenEuclideanMR
>>> testClusteringManhattanMR
>>> testClusteringEuclideanMR
>>> testClusteringManhattanMRWithPayload
>>> testClusteringEuclideanMRWithPayload
>>> testUserDefinedDistanceMeasure
>>>
>>> org.apache.mahout.clustering.meanshift.TestMeanShift
>>> testCanopyEuclideanMRJob
>>>
>>> __________________________________________________
>>> Do You Yahoo!?
>>> En finir avec le spam? Yahoo! Mail vous offre la meilleure protection
>>> possible contre les messages non sollicités
>>> http://mail.yahoo.fr Yahoo! Mail
>>
>> --------------------------
>> Grant Ingersoll
>> http://www.lucidimagination.com
>>
>>
>>
>
--
allenday.skype
+1 (415) 335-4654 (office)
+1 (310) 804-5304 (mobile)
+1 (515) 474-9337 (fax)
Re: OutOfMemory Exception !
Posted by Sean Owen <sr...@gmail.com>.
Yep I saw the same thing. You can either change that settings up your
maximum heap size to, say, 128MB. (Add -Xmx128m to the command line.)
I just committed a change to build.xml to address this. Otherwise the
JVM default is something like 64MB. It's not that your computer is
out of memory.
Yes it just looks like the new default settings assume more memory. It
can be tuned but it seems like 128MB seems like an OK amount of memory
for a worker to use.
Sean
On Mon, Jun 2, 2008 at 8:55 AM, Grant Ingersoll <gs...@apache.org> wrote:
> I'm getting the same errors upon upgrading. We should look at the Hadoop 17
> changes.
>
> On Jun 2, 2008, at 5:50 AM, deneche abdelhakim wrote:
>
>> I checked the last version of Mahout (rev. 662372) and got the following
>> exception with many tests (the list of this tests is at the end of this
>> post):
>>
>> java.io.IOException: Job failed!
>>
>> the following message is printed in System.err :
>>
>> java.lang.OutOfMemoryError: Java heap space
>>
>> I think its somehow caused by using Hadoop 0.17.0 as my own tests run
>> perfectly with Hadoop 0.16.4
>>
>> here are the tests that don't pass
>>
>> org.apache.mahout.clustering.canopy.TestCanopyCreation:
>> testCanopyGenManhattanMR
>> testCanopyGenEuclideanMR
>> testClusteringManhattanMR
>> testClusteringEuclideanMR
>> testClusteringManhattanMRWithPayload
>> testClusteringEuclideanMRWithPayload
>> testUserDefinedDistanceMeasure
>>
>> org.apache.mahout.clustering.meanshift.TestMeanShift
>> testCanopyEuclideanMRJob
>>
>> __________________________________________________
>> Do You Yahoo!?
>> En finir avec le spam? Yahoo! Mail vous offre la meilleure protection
>> possible contre les messages non sollicités
>> http://mail.yahoo.fr Yahoo! Mail
>
> --------------------------
> Grant Ingersoll
> http://www.lucidimagination.com
>
>
>
Re: OutOfMemory Exception !
Posted by Grant Ingersoll <gs...@apache.org>.
Here's the changes: http://hadoop.apache.org/core/docs/r0.17.0/changes.html
http://issues.apache.org/jira/browse/HADOOP-3280 is the only one I
see in a quick glance related to heap size, etc. but I don't think we
are in the streaming case, so not sure.
On Jun 2, 2008, at 8:55 AM, Grant Ingersoll wrote:
> I'm getting the same errors upon upgrading. We should look at the
> Hadoop 17 changes.
>
> On Jun 2, 2008, at 5:50 AM, deneche abdelhakim wrote:
>
>> I checked the last version of Mahout (rev. 662372) and got the
>> following exception with many tests (the list of this tests is at
>> the end of this post):
>>
>> java.io.IOException: Job failed!
>>
>> the following message is printed in System.err :
>>
>> java.lang.OutOfMemoryError: Java heap space
>>
>> I think its somehow caused by using Hadoop 0.17.0 as my own tests
>> run perfectly with Hadoop 0.16.4
>>
>> here are the tests that don't pass
>>
>> org.apache.mahout.clustering.canopy.TestCanopyCreation:
>> testCanopyGenManhattanMR
>> testCanopyGenEuclideanMR
>> testClusteringManhattanMR
>> testClusteringEuclideanMR
>> testClusteringManhattanMRWithPayload
>> testClusteringEuclideanMRWithPayload
>> testUserDefinedDistanceMeasure
>>
>> org.apache.mahout.clustering.meanshift.TestMeanShift
>> testCanopyEuclideanMRJob
>>
>> __________________________________________________
>> Do You Yahoo!?
>> En finir avec le spam? Yahoo! Mail vous offre la meilleure
>> protection possible contre les messages non sollicités
>> http://mail.yahoo.fr Yahoo! Mail
>
> --------------------------
> Grant Ingersoll
> http://www.lucidimagination.com
>
>
--------------------------
Grant Ingersoll
http://www.lucidimagination.com
Re: OutOfMemory Exception !
Posted by Grant Ingersoll <gs...@apache.org>.
I'm getting the same errors upon upgrading. We should look at the
Hadoop 17 changes.
On Jun 2, 2008, at 5:50 AM, deneche abdelhakim wrote:
> I checked the last version of Mahout (rev. 662372) and got the
> following exception with many tests (the list of this tests is at
> the end of this post):
>
> java.io.IOException: Job failed!
>
> the following message is printed in System.err :
>
> java.lang.OutOfMemoryError: Java heap space
>
> I think its somehow caused by using Hadoop 0.17.0 as my own tests
> run perfectly with Hadoop 0.16.4
>
> here are the tests that don't pass
>
> org.apache.mahout.clustering.canopy.TestCanopyCreation:
> testCanopyGenManhattanMR
> testCanopyGenEuclideanMR
> testClusteringManhattanMR
> testClusteringEuclideanMR
> testClusteringManhattanMRWithPayload
> testClusteringEuclideanMRWithPayload
> testUserDefinedDistanceMeasure
>
> org.apache.mahout.clustering.meanshift.TestMeanShift
> testCanopyEuclideanMRJob
>
> __________________________________________________
> Do You Yahoo!?
> En finir avec le spam? Yahoo! Mail vous offre la meilleure
> protection possible contre les messages non sollicités
> http://mail.yahoo.fr Yahoo! Mail
--------------------------
Grant Ingersoll
http://www.lucidimagination.com