memory leak investigations

[Brett Porter <br...@apache.org>]

Hi,

I've done some investigation.

I was able to run the server where it was just checking for non-existant 
updates for some time, and it had no net result on memory usage (it 
constantly goes up, gc's back to the same level, etc).

When I ran a "build all" on cheddar (with no checkouts), net memory 
usage increased 40Mb to build all 130-odd projects (it was at 24mb, now 
it idles at 62 - it was around 80 but 20mb went away after GC). Now, it 
idles at the same level again (even after enqueuing all again, but not 
building).

I took a snapshot at the start and end, and there is 38mb in a single 
hashmap with 513 elements. This map is the 
componentManagersByComponentHashCode field of 
DefaultComponentManagerManager. This would have to be unreleased 
components, right?

Some more things high in the allocation count:
- SecureRunData has 2mb that wasn't there before. It is a possible 
suspect since it is new in 1.0.3. It occurs more than once.
- ContinuumStateContentGenerator has 2mb. Another possible suspect for 
an unreleased component. There are 3 of them in the top 20.
- The second most allocated is a hasmap in the projectBuilder 
(processedProjectCache), at 5mb. This may just be acceptable use.
- DefaultViewContext has much as well, but I think that's because it 
stores all the state generators

The rest seem to be more instances of the above 4.

I don't have more time to investigate - I'd suggest perhaps logging all 
components released when the container shuts down, and just build one 
project from scratch to see what components are left in there at the end..

So it seems the actual specs we need:
- 20mb baseline
- 20mb buffer to build
- caches (eg projects)

It should easily fit in 64mb, with 128mb being safe if we are to spec 
out requirements.

HTH,
Brett

Re: memory leak investigations

[Emmanuel Venisse <em...@venisse.net>]

The memory leak seems to be fixed by my latest patch. Now I have always one statecontentgenerator in 
memory instead of several due to a strange thing with role-hint in components.xml in continuum-web

Emmanuel

Brett Porter a écrit :
> Ok, just watching the telemetry. If I hit "show projects" repeatedly, 
> then I get a steep increase in memory usage, which does not all get 
> collected when I GC. So at about 300k lost per page refresh, that would 
> be part of it, but probably not all (I was refreshing the project list 
> often in the tests below). This could definitely be the 
> statecontentgenerator.
> 
> I next ran a set of builds without doing anything on that screen, and 
> the net result was very little. It seems it could be isolated to the 
> SecureRunData and statecontentgenerator.
> 
> FWIW, if you want to see the snapshots, they are stored under ~bporter 
> on ci.codehaus.org if you have access:
> 
> Launcher-2006-04-12.memory.gz
> Launcher-2006-04-12-2.memory.gz
> 
> They're about 20mb each.
> 
> - Brett
> 
> Brett Porter wrote:
> 
>> Hi,
>>
>> I've done some investigation.
>>
>> I was able to run the server where it was just checking for 
>> non-existant updates for some time, and it had no net result on memory 
>> usage (it constantly goes up, gc's back to the same level, etc).
>>
>> When I ran a "build all" on cheddar (with no checkouts), net memory 
>> usage increased 40Mb to build all 130-odd projects (it was at 24mb, 
>> now it idles at 62 - it was around 80 but 20mb went away after GC). 
>> Now, it idles at the same level again (even after enqueuing all again, 
>> but not building).
>>
>> I took a snapshot at the start and end, and there is 38mb in a single 
>> hashmap with 513 elements. This map is the 
>> componentManagersByComponentHashCode field of 
>> DefaultComponentManagerManager. This would have to be unreleased 
>> components, right?
>>
>> Some more things high in the allocation count:
>> - SecureRunData has 2mb that wasn't there before. It is a possible 
>> suspect since it is new in 1.0.3. It occurs more than once.
>> - ContinuumStateContentGenerator has 2mb. Another possible suspect for 
>> an unreleased component. There are 3 of them in the top 20.
>> - The second most allocated is a hasmap in the projectBuilder 
>> (processedProjectCache), at 5mb. This may just be acceptable use.
>> - DefaultViewContext has much as well, but I think that's because it 
>> stores all the state generators
>>
>> The rest seem to be more instances of the above 4.
>>
>> I don't have more time to investigate - I'd suggest perhaps logging 
>> all components released when the container shuts down, and just build 
>> one project from scratch to see what components are left in there at 
>> the end..
>>
>> So it seems the actual specs we need:
>> - 20mb baseline
>> - 20mb buffer to build
>> - caches (eg projects)
>>
>> It should easily fit in 64mb, with 128mb being safe if we are to spec 
>> out requirements.
>>
>> HTH,
>> Brett
>>
> 
> 
> 

Re: memory leak investigations

[Brett Porter <br...@apache.org>]

Ok, just watching the telemetry. If I hit "show projects" repeatedly, 
then I get a steep increase in memory usage, which does not all get 
collected when I GC. So at about 300k lost per page refresh, that would 
be part of it, but probably not all (I was refreshing the project list 
often in the tests below). This could definitely be the 
statecontentgenerator.

I next ran a set of builds without doing anything on that screen, and 
the net result was very little. It seems it could be isolated to the 
SecureRunData and statecontentgenerator.

FWIW, if you want to see the snapshots, they are stored under ~bporter 
on ci.codehaus.org if you have access:

Launcher-2006-04-12.memory.gz
Launcher-2006-04-12-2.memory.gz

They're about 20mb each.

- Brett

Brett Porter wrote:
> Hi,
> 
> I've done some investigation.
> 
> I was able to run the server where it was just checking for non-existant 
> updates for some time, and it had no net result on memory usage (it 
> constantly goes up, gc's back to the same level, etc).
> 
> When I ran a "build all" on cheddar (with no checkouts), net memory 
> usage increased 40Mb to build all 130-odd projects (it was at 24mb, now 
> it idles at 62 - it was around 80 but 20mb went away after GC). Now, it 
> idles at the same level again (even after enqueuing all again, but not 
> building).
> 
> I took a snapshot at the start and end, and there is 38mb in a single 
> hashmap with 513 elements. This map is the 
> componentManagersByComponentHashCode field of 
> DefaultComponentManagerManager. This would have to be unreleased 
> components, right?
> 
> Some more things high in the allocation count:
> - SecureRunData has 2mb that wasn't there before. It is a possible 
> suspect since it is new in 1.0.3. It occurs more than once.
> - ContinuumStateContentGenerator has 2mb. Another possible suspect for 
> an unreleased component. There are 3 of them in the top 20.
> - The second most allocated is a hasmap in the projectBuilder 
> (processedProjectCache), at 5mb. This may just be acceptable use.
> - DefaultViewContext has much as well, but I think that's because it 
> stores all the state generators
> 
> The rest seem to be more instances of the above 4.
> 
> I don't have more time to investigate - I'd suggest perhaps logging all 
> components released when the container shuts down, and just build one 
> project from scratch to see what components are left in there at the end..
> 
> So it seems the actual specs we need:
> - 20mb baseline
> - 20mb buffer to build
> - caches (eg projects)
> 
> It should easily fit in 64mb, with 128mb being safe if we are to spec 
> out requirements.
> 
> HTH,
> Brett
>