You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Egor Chernodarov <eg...@zarinsk.dem.ru> on 2005/03/25 13:34:22 UTC
java.lang.OutOfMemoryError
Hello!
Periodically, when I try to search, I get "Out of memory" error:
====================================================================================
description The server encountered an internal error () that prevented it from fulfilling this request.
exception
javax.servlet.ServletException
org.apache.jasper.runtime.PageContextImpl.doHandlePageException(PageContextImpl.java:867)
org.apache.jasper.runtime.PageContextImpl.handlePageException(PageContextImpl.java:800)
org.apache.jsp.search_jsp._jspService(search_jsp.java:967)
org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:133)
javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:311)
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:301)
org.apache.jasper.servlet.JspServlet.service(JspServlet.java:248)
javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
root cause
java.lang.OutOfMemoryError
====================================================================================
In index are ~4mln pages.
I start tomcat with these options:
CATALINA_OPTS=-mx1547483648
TOMCAT_OPTS=-mx1547483648
~1.5Gb is not enough for ~4mln index?
How I need to calculate required RAM for stable work?
Also for unknown reason I can not define more than 2Gb RAM for tomcat.
I see
"Error occurred during initialization of VM
Could not reserve enough space for object heap"
But on this server is installed 4Gb RAM.
FreeBSD4.1, JDK1.4.2p6
Thanks!
--
Best regards,
Chernodarov Egor
egor@zarinsk.dem.ru
Re: java.lang.OutOfMemoryError
Posted by Michael Nebel <mi...@nebel.de>.
Hi,
>>>~1.5Gb is not enough for ~4mln index?
it's more than enough! There must be a different problem.
> Stefan Groschupf> As far I know 2GB is the maximum
> Stefan Groschupf> manageable memory for java in 32 bit
> Stefan Groschupf> os systems, isn't it?
>
> It is bad... Somebody tried to use 64 bit JVM and nutch?
> Which problems can arise? Nutch will work on 64 bit JVM?
I think, the 2 GB limit is no nutch, but a jvm problem, so perhaps I can
share some experiences I made the last months. I had to run a
selfwritten loadtest software under java and smashed some several times
against the heap-limit of the jvw.
Linux (Gentoo, amd64, 4 GB RAM)
- Blackdown-jdk 1.4.2.01
-> max 2GB lmit. You can configure more, but it will crash
- Sun-JDK 1.4.2.06 (32bit-Version, no 64 bit version available!!!)
-> ran more stable, but (not suprisingly) the same resul. 2GB max
Sun (Solaris 8, 4 GB-RAM)
- Sun-JDK 1.4.2.06 (inkl. 64 bit patch)
-> no problems at all to use all the 4GB
I would expect, that nutch runs in the same limits.
But does this really matter? As far as I understand the design, I would
try to get many small servers and distribute the load over them. This is
still cheaper than one big server and the main constraint I see is the
needed diskspace. My smallest (and oldest) system is a PIII-800 with 384
MB RAM. It successfull managed to collect an index with about 8 Mio
pages. Then the disk was full :-) Ok, the frontend is seperated and I
have to be patient when updatedb runs, but it works stable and fine ... :-)
regards
Michael
Re[2]: java.lang.OutOfMemoryError
Posted by Gus Bourg <gu...@bourg.net>.
> It is bad... Somebody tried to use 64 bit JVM and nutch?
> Which problems can arise? Nutch will work on 64 bit JVM?
I'm running nutch on a dual opteron box with 4 gigs of ram and sun 64bit
jvm for x86-64 (1.5). It works great.
However, I've seen this same error (java.lang.OutOfMemoryError) but only
once. I restarted nutch and it went away and hasn't been back since (knock
on wood).
Gus
Re[2]: java.lang.OutOfMemoryError
Posted by Egor Chernodarov <eg...@zarinsk.dem.ru>.
Hello, Piotr!
Friday, March 25, 2005, 8:57:02 PM, you wrote:
Piotr Kosiorowski> There is a well known bug -
Piotr Kosiorowski> nutch can throw OutOfMemoryException
Piotr Kosiorowski> if there is not closed double-quote in query:
Piotr Kosiorowski> eg:
Piotr Kosiorowski> "search engine
Piotr Kosiorowski> or
Piotr Kosiorowski> "search engine" "jakarta nutch
Piotr Kosiorowski> I have posted a patch for it on
Piotr Kosiorowski> the list some time ago but it was not
Piotr Kosiorowski> applied - it was for old
Piotr Kosiorowski> "pre-jakarta" package structure - so I will
Piotr Kosiorowski> change it to new package
Piotr Kosiorowski> structure and send again next week.
Piotr Kosiorowski> Please verify if it is the
Piotr Kosiorowski> cause of your OutOfMemory errors.
I do not use double-quote in query.
Simple 1-word query sometimes throw OutOfMemory error.
Piotr Kosiorowski> BTW. I am using 64bit JVM with
Piotr Kosiorowski> nutch without problems (JDK 1.5 - Linux
Piotr Kosiorowski> on Opterons), but I have not
Piotr Kosiorowski> tried to allocate more than 2GB per JVM.
Thanks for information! I will try to install 64 bit JVM. Maybe more
allocation of memory can solve the problem...
But this solution is not like for me ;-/
--
Best regards,
Chernodarov Egor
egor@zarinsk.dem.ru
http://zarinsk.com
Re: java.lang.OutOfMemoryError
Posted by Piotr Kosiorowski <pk...@gmail.com>.
Hello Egor,
There is a well known bug - nutch can throw OutOfMemoryException
if there is not closed double-quote in query:
eg:
"search engine
or
"search engine" "jakarta nutch
I have posted a patch for it on the list some time ago but it was not
applied - it was for old "pre-jakarta" package structure - so I will
change it to new package structure and send again next week.
Please verify if it is the cause of your OutOfMemory errors.
BTW. I am using 64bit JVM with nutch without problems (JDK 1.5 - Linux
on Opterons), but I have not tried to allocate more than 2GB per JVM.
Regards
Piotr
Egor Chernodarov wrote:
> Hello, Stefan!
>
> Friday, March 25, 2005, 7:26:56 PM, вы писали:
>
>
>>>Periodically, when I try to search, I get "Out of memory" error:
>
> Stefan Groschupf> Do you have other webapps in your tomcat?
> No. Only nutch.
>
>
>>>~1.5Gb is not enough for ~4mln index?
>
> Stefan Groschupf> That should be fine, if you have
> Stefan Groschupf> more RAM searching will be just faster.
> That is tomcat should not crash? Even with minimal size of RAM?
>
>
>>>How I need to calculate required RAM for stable work?
>
> Stefan Groschupf> See old wiki hard ware requirements.
> Yeah.. Thanks! I have found it :)
>
> Stefan Groschupf> As far I know 2GB is the maximum
> Stefan Groschupf> manageable memory for java in 32 bit
> Stefan Groschupf> os systems, isn't it?
>
> It is bad... Somebody tried to use 64 bit JVM and nutch?
> Which problems can arise? Nutch will work on 64 bit JVM?
>
>
Re[2]: java.lang.OutOfMemoryError
Posted by Egor Chernodarov <eg...@zarinsk.dem.ru>.
Hello, Stefan!
Friday, March 25, 2005, 7:26:56 PM, вы писали:
>> Periodically, when I try to search, I get "Out of memory" error:
Stefan Groschupf> Do you have other webapps in your tomcat?
No. Only nutch.
>> ~1.5Gb is not enough for ~4mln index?
Stefan Groschupf> That should be fine, if you have
Stefan Groschupf> more RAM searching will be just faster.
That is tomcat should not crash? Even with minimal size of RAM?
>> How I need to calculate required RAM for stable work?
Stefan Groschupf> See old wiki hard ware requirements.
Yeah.. Thanks! I have found it :)
Stefan Groschupf> As far I know 2GB is the maximum
Stefan Groschupf> manageable memory for java in 32 bit
Stefan Groschupf> os systems, isn't it?
It is bad... Somebody tried to use 64 bit JVM and nutch?
Which problems can arise? Nutch will work on 64 bit JVM?
--
Best regards,
Chernodarov Egor
egor@zarinsk.dem.ru
http://zarinsk.com
Re: java.lang.OutOfMemoryError
Posted by Stefan Groschupf <sg...@media-style.com>.
Hi,
> Periodically, when I try to search, I get "Out of memory" error:
Do you have other webapps in your tomcat?
> ~1.5Gb is not enough for ~4mln index?
>
That should be fine, if you have more RAM searching will be just faster.
> How I need to calculate required RAM for stable work?
See old wiki hard ware requirements.
> Also for unknown reason I can not define more than 2Gb RAM for tomcat.
>
As far I know 2GB is the maximum manageable memory for java in 32 bit
os systems, isn't it?
HTH
Stefan