You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Egor Chernodarov <eg...@zarinsk.dem.ru> on 2005/03/25 13:34:22 UTC

java.lang.OutOfMemoryError

Hello!

Periodically, when I try to search, I get "Out of memory" error:

====================================================================================
description The server encountered an internal error () that prevented it from fulfilling this request.
exception
javax.servlet.ServletException
        org.apache.jasper.runtime.PageContextImpl.doHandlePageException(PageContextImpl.java:867)
        org.apache.jasper.runtime.PageContextImpl.handlePageException(PageContextImpl.java:800)
        org.apache.jsp.search_jsp._jspService(search_jsp.java:967)
        org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:133)
        javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
        org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:311)
        org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:301)
        org.apache.jasper.servlet.JspServlet.service(JspServlet.java:248)
        javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
root cause
java.lang.OutOfMemoryError
====================================================================================

In index are ~4mln pages.
I start tomcat with these options:
  CATALINA_OPTS=-mx1547483648
  TOMCAT_OPTS=-mx1547483648

~1.5Gb is not enough for ~4mln index?
How I need to calculate required RAM for stable work?

Also for unknown reason I can not define more than 2Gb RAM for tomcat.
I see
"Error occurred during initialization of VM
Could not reserve enough space for object heap"
But on this server is installed 4Gb RAM.

FreeBSD4.1, JDK1.4.2p6


Thanks!

-- 
Best regards,
 Chernodarov Egor
 egor@zarinsk.dem.ru
                                


Re: java.lang.OutOfMemoryError

Posted by Michael Nebel <mi...@nebel.de>.
Hi,

>>>~1.5Gb is not enough for ~4mln index?

it's more than enough! There must be a different problem.

> Stefan Groschupf> As far I know 2GB is the maximum
> Stefan Groschupf> manageable memory for java in 32 bit 
> Stefan Groschupf> os systems, isn't it?
> 
> It is bad...  Somebody tried to use 64 bit JVM and nutch?
> Which problems can arise?  Nutch will work on 64 bit JVM?

I think, the 2 GB limit is no nutch, but a jvm problem, so perhaps I can 
share some experiences I made the last months. I had to run  a 
selfwritten loadtest software under java and smashed some several times 
against the heap-limit of the jvw.

Linux (Gentoo, amd64, 4 GB RAM)

- Blackdown-jdk 1.4.2.01
-> max 2GB lmit. You can configure more, but it will crash

- Sun-JDK 1.4.2.06 (32bit-Version, no 64 bit version available!!!)
-> ran more stable, but (not suprisingly) the same resul. 2GB max

Sun (Solaris 8, 4 GB-RAM)

- Sun-JDK 1.4.2.06 (inkl. 64 bit patch)
-> no problems at all to use all the 4GB

I would expect, that nutch runs in the same limits.

But does this really matter?  As far as I understand the design, I would 
try to get many small servers and distribute the load over them. This is 
still cheaper than one big server and the main constraint I see is the 
needed diskspace. My smallest (and oldest) system is a PIII-800 with 384 
MB RAM. It successfull managed to collect an index with about 8 Mio 
pages. Then the disk was full :-) Ok, the frontend is seperated and I 
have to be patient when updatedb runs, but it works stable and fine ... :-)

regards

	Michael


Re[2]: java.lang.OutOfMemoryError

Posted by Gus Bourg <gu...@bourg.net>.
> It is bad...  Somebody tried to use 64 bit JVM and nutch?
> Which problems can arise?  Nutch will work on 64 bit JVM?

I'm running nutch on a dual opteron box with 4 gigs of ram and sun 64bit 
jvm for x86-64 (1.5). It works great.

However, I've seen this same error (java.lang.OutOfMemoryError) but only 
once. I restarted nutch and it went away and hasn't been back since (knock 
on wood).

Gus

Re[2]: java.lang.OutOfMemoryError

Posted by Egor Chernodarov <eg...@zarinsk.dem.ru>.
Hello, Piotr!

Friday, March 25, 2005, 8:57:02 PM, you wrote:

Piotr Kosiorowski> There is a well known bug -
Piotr Kosiorowski> nutch can throw OutOfMemoryException
Piotr Kosiorowski> if there is not closed double-quote in query:
Piotr Kosiorowski> eg:
Piotr Kosiorowski> "search engine
Piotr Kosiorowski> or
Piotr Kosiorowski> "search engine" "jakarta nutch
Piotr Kosiorowski> I have posted a patch for it on
Piotr Kosiorowski> the list some time ago but it was not 
Piotr Kosiorowski> applied - it was for old
Piotr Kosiorowski> "pre-jakarta" package structure - so I will 
Piotr Kosiorowski> change it to new package
Piotr Kosiorowski> structure and send again next week.
Piotr Kosiorowski> Please verify if it is the
Piotr Kosiorowski> cause of your OutOfMemory errors.
I do not use double-quote in query.
Simple 1-word query sometimes throw OutOfMemory error.

Piotr Kosiorowski> BTW. I am using 64bit JVM with
Piotr Kosiorowski> nutch without problems (JDK 1.5 - Linux 
Piotr Kosiorowski> on Opterons), but I have not
Piotr Kosiorowski> tried to allocate more than 2GB per JVM.
Thanks for information! I will try to install 64 bit JVM. Maybe more
allocation of memory can solve the problem...
But this solution is not like for me ;-/


-- 
Best regards,               
 Chernodarov Egor                          
 egor@zarinsk.dem.ru
 http://zarinsk.com
        


Re: java.lang.OutOfMemoryError

Posted by Piotr Kosiorowski <pk...@gmail.com>.
Hello Egor,
There is a well known bug - nutch can throw OutOfMemoryException
if there is not closed double-quote in query:
eg:
"search engine
or
"search engine" "jakarta nutch
I have posted a patch for it on the list some time ago but it was not 
applied - it was for old "pre-jakarta" package structure - so I will 
change it to new package structure and send again next week.
Please verify if it is the cause of your OutOfMemory errors.
BTW. I am using 64bit JVM with nutch without problems (JDK 1.5 - Linux 
on Opterons), but I have not tried to allocate more than 2GB per JVM.

Regards
Piotr



Egor Chernodarov wrote:
> Hello, Stefan!
> 
> Friday, March 25, 2005, 7:26:56 PM, вы писали:
> 
> 
>>>Periodically, when I try to search, I get "Out of memory" error:
> 
> Stefan Groschupf> Do you have other webapps in your tomcat?
> No. Only nutch.
> 
> 
>>>~1.5Gb is not enough for ~4mln index?
> 
> Stefan Groschupf> That should be fine, if you have
> Stefan Groschupf> more RAM searching will be just faster.
> That is tomcat should not crash? Even with minimal size of RAM?
> 
> 
>>>How I need to calculate required RAM for stable work?
> 
> Stefan Groschupf> See old wiki hard ware requirements.
> Yeah.. Thanks! I have found it :)
> 
> Stefan Groschupf> As far I know 2GB is the maximum
> Stefan Groschupf> manageable memory for java in 32 bit 
> Stefan Groschupf> os systems, isn't it?
> 
> It is bad...  Somebody tried to use 64 bit JVM and nutch?
> Which problems can arise?  Nutch will work on 64 bit JVM?
> 
> 



Re[2]: java.lang.OutOfMemoryError

Posted by Egor Chernodarov <eg...@zarinsk.dem.ru>.
Hello, Stefan!

Friday, March 25, 2005, 7:26:56 PM, вы писали:

>> Periodically, when I try to search, I get "Out of memory" error:
Stefan Groschupf> Do you have other webapps in your tomcat?
No. Only nutch.

>> ~1.5Gb is not enough for ~4mln index?
Stefan Groschupf> That should be fine, if you have
Stefan Groschupf> more RAM searching will be just faster.
That is tomcat should not crash? Even with minimal size of RAM?

>> How I need to calculate required RAM for stable work?
Stefan Groschupf> See old wiki hard ware requirements.
Yeah.. Thanks! I have found it :)

Stefan Groschupf> As far I know 2GB is the maximum
Stefan Groschupf> manageable memory for java in 32 bit 
Stefan Groschupf> os systems, isn't it?

It is bad...  Somebody tried to use 64 bit JVM and nutch?
Which problems can arise?  Nutch will work on 64 bit JVM?


-- 
Best regards,               
 Chernodarov Egor                          
 egor@zarinsk.dem.ru
 http://zarinsk.com
        


Re: java.lang.OutOfMemoryError

Posted by Stefan Groschupf <sg...@media-style.com>.
Hi,

> Periodically, when I try to search, I get "Out of memory" error:

Do you have other webapps in your tomcat?
> ~1.5Gb is not enough for ~4mln index?
>
That should be fine, if you have more RAM searching will be just faster.

> How I need to calculate required RAM for stable work?
See old wiki hard ware requirements.

> Also for unknown reason I can not define more than 2Gb RAM for tomcat.
>
As far I know 2GB is the maximum manageable memory for java in 32 bit 
os systems, isn't it?

HTH
Stefan