You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Deals Collect <de...@gmail.com> on 2013/04/09 01:49:08 UTC
Permgen size keeps increasing
Hi all,
I have a site crawling the deals every days. I have a problem with permgen
size. It just keeps increasing after each crawling. That's why I got the
"Out of memory" issue. As I know, Permgen heap just stores the class
definition, metadata... I'm not sure why it keeps increasing like this...
I'm using Nutch 1.4 and Solr 3.1. Here is the link that you can see the
permgen size: http://www.dealscollect.com.au/permgen It is the last one:
Name: PS Perm Gen
Type: Non-heap memory
Max Usage: 268435456
Used: 75770136
Many thanks,
Vu
Re: Permgen size keeps increasing
Posted by Sebastian Nagel <wa...@googlemail.com>.
Hi,
> It just keeps increasing after each crawling.
What does this precisely mean?
(a) Are you running one crawl process with many cycles (depth)
by launching "bin/nutch crawl" (org.apache.nutch.crawl.Crawl)
(b) or in separate steps (inject, generate, fetch, parse, updatedb, ...)?
For (a) see NUTCH-356 and NUTCH-1087 which may cause the problem.
Sebastian
On 04/09/2013 01:49 AM, Deals Collect wrote:
> Hi all,
>
> I have a site crawling the deals every days. I have a problem with permgen
> size. It just keeps increasing after each crawling. That's why I got the
> "Out of memory" issue. As I know, Permgen heap just stores the class
> definition, metadata... I'm not sure why it keeps increasing like this...
> I'm using Nutch 1.4 and Solr 3.1. Here is the link that you can see the
> permgen size: http://www.dealscollect.com.au/permgen It is the last one:
>
> Name: PS Perm Gen
> Type: Non-heap memory
> Max Usage: 268435456
> Used: 75770136
>
> Many thanks,
> Vu
>