Memory usage question

[Robert Bartlett <rb...@digitalphx.com>]

I remember someone saying something about memory usage per email that
spamd uses to scan? But cannot find the email, what is the estimated
amount of memory used per SA scan? I also have clamav set up.
 
Thank you
Robert Bartlett
Director of Software Engineering
Digital Phoenix Hosting & Design
http://www.digitalphx.com
"For Your Digital Existence"
 

Re[2]: Memory usage question

[Robert Menschel <Ro...@Menschel.net>]

Hello Chr.,

Friday, September 17, 2004, 7:58:11 AM, Stucki (Christoph von Stuckrad)
wrote: 

CvS> On Fri, Sep 17, 2004 at 10:42:20AM -0400, Matt Kettler wrote:
>> Since your box has 256mb of physical ram, I'd limit it to maximum of 
>> 256mb/15mb = 17 spamd's at the highest. I'd really suggest using something
>> much lower like 10 unless you add some ram.

CvS> So a system with 1G Memory (+2GSwap) DualPentium4 simply stopped
CvS> completely just by crowding its space with max 32 copies of spamd
CvS> (each forking with near 50M). The system was to slow to reboot
CvS> correctly and had to be 'reset' and fsck-ed.

CvS> So you have to watch closely, if the system is as small as the
CvS> above. Better invest in lots of memory...

However, for a system that gets only
RB>> At one point we hit 60 emails in a span of 5 minutes.
-m of 4 or 5 is probably adequate, in which case 256 Meg may be
sufficient, if slow.

(Me, I use a 1 Gig Windows machine just for word processing, web page
editing, and the occasional mass-check. Norton Utilities normally reports
50% in use, no matter what I do.)

Bob Menschel

Re: Memory usage question

["Chr. von Stuckrad" <st...@mi.fu-berlin.de>]

On Fri, Sep 17, 2004 at 10:42:20AM -0400, Matt Kettler wrote:
> Since your box has 256mb of physical ram, I'd limit it to maximum of 
> 256mb/15mb = 17 spamd's at the highest. I'd really suggest using something 
> much lower like 10 unless you add some ram.

Even this seems to be dangerous (sometimes).

We just had a crash of the spamd-server,
seemingly by being hit with lots of
maximally large mails at the same time.

So a system with 1G Memory (+2GSwap)
DualPentium4 simply stopped completely
just by crowding its space with max 32
copies of spamd (each forking with
near 50M). The system was to slow to
reboot correctly and had to be 'reset'
and fsck-ed.

So you have to watch closely, if the system
is as small as the above. Better invest in
lots of memory...

Stucki


-- 
Christoph von Stuckrad     * * |nickname |<st...@math.fu-berlin.de>\
Freie Universitaet Berlin  |/_*|'stucki' |Tel(days):+49 30 838-75 459|
Fachbereich Mathematik, EDV|\ *|if online|Tel(else):+49 30 77 39 6600|
Arnimallee 2-6/14195 Berlin* * |on IRCnet|Fax(alle):+49 30 838-75454/

RE: Memory usage question

[Matt Kettler <mk...@evi-inc.com>]

At 09:23 AM 9/17/2004, Robert Bartlett wrote:
>Are you using the -m parameter of spamd to limit the number of children
>it
>will spawn? I'd suggest something like -m 6 to start with.
>
>
>Yeah it is setup for 50:
>
>-d -c -a -m50 -u user -v -H

50 is a LOT of spamd's... even at the low-end of 15mb each that's 750mb of 
memory allocation.

Since your box has 256mb of physical ram, I'd limit it to maximum of 
256mb/15mb = 17 spamd's at the highest. I'd really suggest using something 
much lower like 10 unless you add some ram.

RE: Memory usage question

[Robert Bartlett <rb...@digitalphx.com>]

-----Original Message-----
From: Matt Kettler [mailto:mkettler_sa@comcast.net] 
Sent: Friday, September 17, 2004 6:12 AM
To: rbartlett@digitalphx.com; users@spamassassin.apache.org
Subject: RE: Memory usage question

At 06:56 PM 9/16/2004 -0700, Robert Bartlett wrote:
>Thanks for the reply! Here is the deal, we are currently deciding what
>we want to do next. Currently we have a Celeron 2.4 gig system with 256
>megs of ram and a 40 gig hdd. In the past week or so our system has
come
>to a halt, under 3 megs available, due to a bunch of emails coming in
at
>once. At one point we hit 60 emails in a span of 5 minutes.

Are you using the -m parameter of spamd to limit the number of children
it 
will spawn? I'd suggest something like -m 6 to start with.


Yeah it is setup for 50:

-d -c -a -m50 -u user -v -H

-d, --daemonize                    Daemonize
-c, --create-prefs                 Create user preferences files
-a, --auto-whitelist, --whitelist  Use auto-whitelists
-u username, --username=username   Run as username
-v, --vpopmail                     Enable vpopmail config
-H dir                             Specify a different HOME directory,
path optional

RE: Memory usage question

[Matt Kettler <mk...@comcast.net>]

At 06:56 PM 9/16/2004 -0700, Robert Bartlett wrote:
>Thanks for the reply! Here is the deal, we are currently deciding what
>we want to do next. Currently we have a Celeron 2.4 gig system with 256
>megs of ram and a 40 gig hdd. In the past week or so our system has come
>to a halt, under 3 megs available, due to a bunch of emails coming in at
>once. At one point we hit 60 emails in a span of 5 minutes.

Are you using the -m parameter of spamd to limit the number of children it 
will spawn? I'd suggest something like -m 6 to start with.

Re: Memory usage question

[Kai Schaetzl <ma...@conactive.com>]

Matt Kettler wrote on Thu, 16 Sep 2004 21:43:01 -0400:

> Chris S reported his spamd swelling to 45mb with a huge version of 
> bigevil.cf he was testing.
>

The latest bigevil.cf needs about 40 - 50 MB *alone*! Together with 
several SARE rules our spamd processes were around 90 MB lately. That's 
where I removed bigevil (and activated SURBL on 3.0) and that other big 
ruleset (don't remember the name at the moment), so that our spamd prcess 
is now about 50 MB. I didn't see much difference in detection on the 2.63 
systems, so bigevil is simply not worth the huge memory consumption.


Kai

-- 

Kai Schätzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com
IE-Center: http://ie5.de & http://msie.winware.org

Re: Memory usage question

[Loren Wilton <lw...@earthlink.net>]

> Thanks for the reply! Here is the deal, we are currently deciding what
> we want to do next. Currently we have a Celeron 2.4 gig system with 256
> megs of ram and a 40 gig hdd. In the past week or so our system has come
> to a halt, under 3 megs available, due to a bunch of emails coming in at
> once. At one point we hit 60 emails in a span of 5 minutes. It is a

You don't sound like you have a huge mail rate.  But if you are using addon
rules, you may be getting in the area of doubling or more the original
number of rules, so you should probably at least double the memory
requirement per spamassassin process (or process driver).

My suggestion is that that machine is plenty fast enough for you, but I
would at least double the memory on it.  If its cheap I'd take it up to a
gig or more and be done with it.

        Loren

RE: Memory usage question

[Robert Bartlett <rb...@digitalphx.com>]

-----Original Message-----
From: Matt Kettler [mailto:mkettler@evi-inc.com] 
Sent: Thursday, September 16, 2004 6:43 PM
To: rbartlett@digitalphx.com; users@spamassassin.apache.org
Subject: Re: Memory usage question

At 09:26 PM 9/16/2004, Robert Bartlett wrote:
>I remember someone saying something about memory usage per email that 
>spamd uses to scan? But cannot find the email, what is the estimated 
>amount of memory used per SA scan? I also have clamav set up

Varies a lot depending on your configuration (bayes vs no bayes, add on 
rules, etc).

If I start spamd on my system (don't normaly use it because I use 
MailScanner which calls the API directly) it pops up with a RSS of 26mb.
I 
use bayes with an enlarged database size (200k tokens, instead of 150k)
, 
and a few add-on rules.

A 200k token bayes db should be about 10mb based on info in the manpage,
so 
disabling bayes and using only stock rules could take spamd down to as 
little as 15mb, however, I've not got the ability to test that right
now.

Chris S reported his spamd swelling to 45mb with a huge version of 
bigevil.cf he was testing.

Thanks for the reply! Here is the deal, we are currently deciding what
we want to do next. Currently we have a Celeron 2.4 gig system with 256
megs of ram and a 40 gig hdd. In the past week or so our system has come
to a halt, under 3 megs available, due to a bunch of emails coming in at
once. At one point we hit 60 emails in a span of 5 minutes. It is a
system we are "renting" at a colo. So Im heading this project up trying
to decide to either rent out cabinet space and build our own systems and
do it that way, or just upgrade the current system. Currently we are
running Fedora Core 1 with clamav. I do not believe we have bayes
running, I assume we don't since I do not know how this would be set up.

We also use Rules De Jour with all rules available except Big Evil, we
are using RBL. I know that when I restart spamd it shows this:

99.9  9.5 29068 24300 (24300 being RSS)

I also use vpopmail for virtual domain setup.

Re: Memory usage question

[Matt Kettler <mk...@evi-inc.com>]

At 09:26 PM 9/16/2004, Robert Bartlett wrote:
>I remember someone saying something about memory usage per email that 
>spamd uses to scan? But cannot find the email, what is the estimated 
>amount of memory used per SA scan? I also have clamav set up

Varies a lot depending on your configuration (bayes vs no bayes, add on 
rules, etc).

If I start spamd on my system (don't normaly use it because I use 
MailScanner which calls the API directly) it pops up with a RSS of 26mb.  I 
use bayes with an enlarged database size (200k tokens, instead of 150k) , 
and a few add-on rules.

A 200k token bayes db should be about 10mb based on info in the manpage, so 
disabling bayes and using only stock rules could take spamd down to as 
little as 15mb, however, I've not got the ability to test that right now.

Chris S reported his spamd swelling to 45mb with a huge version of 
bigevil.cf he was testing.