You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@forrest.apache.org by Thorsten Scherler <th...@apache.org> on 2008/07/31 00:01:23 UTC
memory leaks in forrest
Hi all,
lately I saw a couple of OutOfMemoryError reports around forrest. Brain
and Gavin reported them around the dispatcher plugin. Actually today I
ran to an OutOfMemoryError within our solr plugin.
The code is not used with the dispatcher but directly. I am using
http://svn.apache.org/viewvc/forrest/trunk/whiteboard/plugins/org.apache.forrest.plugin.output.solr/src/java/org/apache/forrest/solr/client/SolrSearchGenerator.java?view=log
Doing a "top" on the box I see that the memory never got cleaned up. It
keeps on growing till it reaches the max. You can see it quite fast in
limiting the max memory to a real small amount (e.g. 128M) the memory
usage is keep on growing till forrest fails with Heaps size error.
My first impression would be that is caused by the above generator but
seeing all the recent threads about similar situations it makes me
wander whether that may be a general problem.
Does anybody see an obvious leak in the above generator? Does anybody
has an idea how we can trigger the leak?
TIA
salu2
--
Thorsten Scherler thorsten.at.apache.org
Open Source Java consulting, training and solutions
Re: memory leaks in forrest
Posted by Thorsten Scherler <th...@juntadeandalucia.es>.
On Thu, 2008-07-31 at 11:47 +0100, Ross Gardler wrote:
> David Crossley wrote:
> > Thorsten Scherler wrote:
>
> ...
>
> > Aside: Oh it would be blissful to get our internal
> > Cocoon upgraded to at least use Cocoon-2.1 head.
> > Ralph Goers has fixed that problem which causes FOR-591.
>
> Agreed. I think this will solve quite a few problems.
>
> Thorsten, what is the status of roids in the labs? I started looking at
> it the other day but ran out of time (as usual).
It is getting pretty much in shape and I am using it in production in my
current project. The multi-threaded part is the real awesome part. ;)
There are some small things missing (saving css, images, ... to the
export) but as soon I close
https://issues.apache.org/jira/browse/LABS-118 it is ready for prime
time.
salu2
>
> Ross
--
Thorsten Scherler thorsten.at.apache.org
Open Source Java consulting, training and solutions
Re: memory leaks in forrest
Posted by Ross Gardler <rg...@apache.org>.
David Crossley wrote:
> Thorsten Scherler wrote:
...
> Aside: Oh it would be blissful to get our internal
> Cocoon upgraded to at least use Cocoon-2.1 head.
> Ralph Goers has fixed that problem which causes FOR-591.
Agreed. I think this will solve quite a few problems.
Thorsten, what is the status of roids in the labs? I started looking at
it the other day but ran out of time (as usual).
Ross
Re: memory leaks in forrest
Posted by Thorsten Scherler <th...@juntadeandalucia.es>.
On Thu, 2008-07-31 at 12:34 +1000, David Crossley wrote:
> Thorsten Scherler wrote:
> >
> > lately I saw a couple of OutOfMemoryError reports around forrest. Brain
> > and Gavin reported them around the dispatcher plugin. Actually today I
> > ran to an OutOfMemoryError within our solr plugin.
> >
> > The code is not used with the dispatcher but directly. I am using
> > http://svn.apache.org/viewvc/forrest/trunk/whiteboard/plugins/org.apache.forrest.plugin.output.solr/src/java/org/apache/forrest/solr/client/SolrSearchGenerator.java?view=log
> >
> > Doing a "top" on the box I see that the memory never got cleaned up. It
> > keeps on growing till it reaches the max. You can see it quite fast in
> > limiting the max memory to a real small amount (e.g. 128M) the memory
> > usage is keep on growing till forrest fails with Heaps size error.
> >
> > My first impression would be that is caused by the above generator but
> > seeing all the recent threads about similar situations it makes me
> > wander whether that may be a general problem.
>
> I don't use the solr plugin.
>
> I do see the issue with one of my complex sites that
> uses dispatcher plugins (not your other whiteboard work).
> Also with the forrest/site-author site which is skins-based.
>
> The memory usage does not increase quickly, as for yours,
> but yes it does increase. Only manual tests so not yet
> reached a crash.
>
> However, this seems to be consistent with the bugs that
> we have already identified.
> https://issues.apache.org/jira/browse/FOR-591
> "Memory Leak with XMLFileModule"
> and there was some other known dispatcher issues.
>
Thank you very much David to point me to this issue. I will now
investigate whether this can be the cause (but it really sounds like it)
and see how we can fix this.
> Aside: Oh it would be blissful to get our internal
> Cocoon upgraded to at least use Cocoon-2.1 head.
> Ralph Goers has fixed that problem which causes FOR-591.
I first need to find a quick workaround for the project based on
forrest in my work (need to deliver till Monday a new version of this
work project), but then I need to fix this issue once and for all in
forrest and will try to update our cocoon. I personally prefer to 2.2
but I fear the pass-through feature (missing in 2.2. AFAIK and
extensively used in forrest) will be a showstopper but 2.1 should work
just fine.
>
> > Does anybody see an obvious leak in the above generator? Does anybody
> > has an idea how we can trigger the leak?
>
> I have not looked - perhaps someone else can.
>
> Does the solr plugin make extensive use of XML File Module?
> You might reach crisis more quickly.
>
> Also see Tim's comments in FOR-591 regarding perhaps
> "compounded this issue with similar simple caching in the locationmap".
> Maybe that part has already been addressed.
> There was recently some other locationmap changes.
>
> Dunno, sorry, just some ideas.
Thank you very much for your thoughts. As usual most valuable.
salu2
>
> -David
--
Thorsten Scherler thorsten.at.apache.org
Open Source Java consulting, training and solutions
Re: memory leaks in forrest
Posted by David Crossley <cr...@apache.org>.
Thorsten Scherler wrote:
>
> lately I saw a couple of OutOfMemoryError reports around forrest. Brain
> and Gavin reported them around the dispatcher plugin. Actually today I
> ran to an OutOfMemoryError within our solr plugin.
>
> The code is not used with the dispatcher but directly. I am using
> http://svn.apache.org/viewvc/forrest/trunk/whiteboard/plugins/org.apache.forrest.plugin.output.solr/src/java/org/apache/forrest/solr/client/SolrSearchGenerator.java?view=log
>
> Doing a "top" on the box I see that the memory never got cleaned up. It
> keeps on growing till it reaches the max. You can see it quite fast in
> limiting the max memory to a real small amount (e.g. 128M) the memory
> usage is keep on growing till forrest fails with Heaps size error.
>
> My first impression would be that is caused by the above generator but
> seeing all the recent threads about similar situations it makes me
> wander whether that may be a general problem.
I don't use the solr plugin.
I do see the issue with one of my complex sites that
uses dispatcher plugins (not your other whiteboard work).
Also with the forrest/site-author site which is skins-based.
The memory usage does not increase quickly, as for yours,
but yes it does increase. Only manual tests so not yet
reached a crash.
However, this seems to be consistent with the bugs that
we have already identified.
https://issues.apache.org/jira/browse/FOR-591
"Memory Leak with XMLFileModule"
and there was some other known dispatcher issues.
Aside: Oh it would be blissful to get our internal
Cocoon upgraded to at least use Cocoon-2.1 head.
Ralph Goers has fixed that problem which causes FOR-591.
> Does anybody see an obvious leak in the above generator? Does anybody
> has an idea how we can trigger the leak?
I have not looked - perhaps someone else can.
Does the solr plugin make extensive use of XML File Module?
You might reach crisis more quickly.
Also see Tim's comments in FOR-591 regarding perhaps
"compounded this issue with similar simple caching in the locationmap".
Maybe that part has already been addressed.
There was recently some other locationmap changes.
Dunno, sorry, just some ideas.
-David