Occasial extreme memory consumption during parse

[Markus Jelsma <ma...@openindex.io>]

Hi,

Anyone knows what can cause the parsers to blow up RAM consumption? This is 
about a server (runnning locally) with 1GB RAM and 600MB heap space allocated. 
I know, it's not much for the OS but everything usually runs fine.

But, during yet another very large crawl there are some peaks during the parse 
in which memory consumption gets out of control. Almost immediately the server 
begins thrashing, a typical swap in/out graphs shows it clearly. Suddenly it 
seems to resolve itself and resume normal parse.

Any idea where i've to look for? Any pointers?

Thanks