Re: Weekly run ...

[Justin Mason <jm...@jmason.org>]

Theo Van Dinter writes:
> 432243 runs fine, but 432244 causes OOMs.  I poked around some more to
> determine the other keys that need to be deleted, and was going to just
> add them in, but I got to thinking:  Since any plugin can add in data to
> PMS, there can be any number of circular refs that we don't know about, so
> deleting all of the keys is the only way we can be sure to clear them out.
> 
> FWIW: I found resolver and uribl_scanstate which needed to be removed, but
> even then I still got OOMs, it just took longer.  I assume there are other
> net-test-related things that I'm somehow not seeing via Data::Dumper.
> 
> I'm going to commit in a change that changes PMS::finish() to do:
> 
> %{$self} = ();
> 
> which ends up deleting all the keys while not requiring the foreach() method
> that we used previous, so I hope this solves JM's original issue if that being
> inefficient.

Yep, looks fine to me!  Good catch.

> I do think that we still have leaks elsewhere since the process memory
> continues to grow, but it's much lower than before.  I'll keep digging.

Please do...

Last time I looked, there were some tiny leaks over time in the perl
interpreter, but nothing to be worried about (since we restart
every N messages in spamd and mass-check).

--j.

Re: Weekly run ...

[Theo Van Dinter <fe...@apache.org>]

On Sun, Sep 03, 2006 at 11:13:46PM +0100, Justin Mason wrote:
> > I do think that we still have leaks elsewhere since the process memory
> > continues to grow, but it's much lower than before.  I'll keep digging.
> 
> Please do...
> 
> Last time I looked, there were some tiny leaks over time in the perl
> interpreter, but nothing to be worried about (since we restart
> every N messages in spamd and mass-check).

Well, that's not a reason to not be worried about them. :)    Not everyone
restarts their processes, so it's a problem we should deal with if we can.

The more I look at things, the more I believe there's a major leak
related to the URIDNSBL plugin.  Unfortunately, I can no longer figure
out how that code works due to the complexity.  I've spent a good amount
of time looking at it and find that it's too confusing to follow.

In short, if I run just SURBL rules, I OOM.  If I run other network
tests (say, RCVD_IN_XBL,) I don't.  If I don't run net rules at all, I
don't OOM.  Therefore it seems as if the plugin is the issue.  I noticed
that URIDNSBL is the only thing using the AsyncLoop code, so there could
be a problem in there instead/as well.

Justin, since you were the one who put in AsyncLoop, can you poke around and
figure what's causing the issue?

-- 
Randomly Generated Tagline:
"Sorry, not tonight.  I have to floss my cat." - Random Turn-down line