You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Simon Loewenthal <si...@klunky.co.uk> on 2013/03/06 15:17:35 UTC
Re: Upgrade from SA 3.3.1 to 3.3.2 - increase in memory requirements on Debian 6
Hi KAM,
Options are : /usr/sbin/spamd --create-prefs -x -q --ipv4
--max-children 1 --timeout-child 180 --sql-config --nouser-config
--username spamd --helper-home-dir -s /var/log/spamd.log
--virtual-config-dir=/users/%d/%u -d --pidfile=/var/run/spamd.pid
( 1
child set because of lack of memory. 2 causes it to swap)
As far as I
can tell no rulesets have changed. I have these additional ones added :
# MALWARE BLOCKLIST
"http://www.malwarepatrol.net/cgi/submit?action=list_sa" -O
MALWARE.BLOCKLIST.CF
# 99_ANONWHOIS
"http://anonwhois.org/99_anonwhois.cf" -O 99_ANONWHOIS.CF
# kAOS
rESIgns MTX Blacklist
/usr/local/bin/mtx_blacklist.pl
# SOUGHT rules
via sa-update
--channel SOUGHT.rules.yerp.org --channel
updates.spamassassin.org
# Generate Spamassassin rules from the
phishing_reply_addresses list
/usr/local/bin/addresses2spamassassin.pl
- Prodcues file: LOCAL_PHISHING_REPLY.CF
CUSTOM RULES IN LOCAL.CF
I
have a several custom rules comprising of 231 lines. These are simple
rules comprising of some simple regex. Some were copied from this
mailing list. I should turn these off and see what happens.
In the
past 24 hours the spamd memory usage has dropped to 198 Mb, which is a
relief, but this happened after I did a update on the server from
squeeze/updates, squeeze, and security. Before time I just had security
configured.
Cheers, S
On 2013-03-06 14:57, Kevin A. McGrail wrote:
> On 3/5/2013 7:36 AM, Simon Loewenthal wrote:
>
>> I just upgraded
a small server from 3.3.1 to 3.3.2 (Debain Squeeze).
>>
>> I notice
that spamd now takes 64% of the memory which is 317 mb. This is rather
high in my opinion.
>>
>> I realize this may well be a Debian specific
question, but does _spamassassin 3.3.2-2~bpo60+1_ have any performance
related dependencies on other packages? Currently, Perl is
5.10.1-17squeeze4.
> What spamd options are you using now? What were you
using?
>
> What rulesets are you using now? What were you using?
>
>
Regards,
> KAM
Re: Upgrade from SA 3.3.1 to 3.3.2 - increase in memory requirements
on Debian 6
Posted by Axb <ax...@gmail.com>.
On 03/06/2013 03:17 PM, Simon Loewenthal wrote:
>
>
> Hi KAM,
>
> Options are : /usr/sbin/spamd --create-prefs -x -q --ipv4
> --max-children 1 --timeout-child 180 --sql-config --nouser-config
> --username spamd --helper-home-dir -s /var/log/spamd.log
> --virtual-config-dir=/users/%d/%u -d --pidfile=/var/run/spamd.pid
>
> ( 1
> child set because of lack of memory. 2 causes it to swap)
>
> As far as I
> can tell no rulesets have changed. I have these additional ones added :
>
>
> # MALWARE BLOCKLIST
>
> "http://www.malwarepatrol.net/cgi/submit?action=list_sa" -O
> MALWARE.BLOCKLIST.CF
got any hits on this to make it worthwhile?
> # 99_ANONWHOIS
> "http://anonwhois.org/99_anonwhois.cf" -O 99_ANONWHOIS.CF
you can probably loose this - it was hand compiled and low hit rate.
(is it really still alive??
> # Generate Spamassassin rules from the
> phishing_reply_addresses list
> /usr/local/bin/addresses2spamassassin.pl
> - Prodcues file: LOCAL_PHISHING_REPLY.CF
iirc, this ruleset is HUGE and I honestly doubt you see many hist with
it which dont' overlap with other stuff.
And last but not least - if 317 mb is 64% of your total memory: add RAM
to your box
Axb
Re: Upgrade from SA 3.3.1 to 3.3.2 - increase in memory requirements on Debian 6
Posted by JK4 <ju...@klunky.co.uk>.
I have Razor and Pyzor running.
(Although Pyzor seems to give
errors these days with a "pyzor: error: TERMINATED, signal 15 (000f)"
even though .pyzor/servers is in the right place, and pyzor discover
downloads the server(s) correctly. Rather strange. )
Never used
iXhash. Shall look into this on
http://wiki.apache.org/spamassassin/iXhash
Cheers, S
On 2013-03-06
16:11, Axb wrote:
> On 03/06/2013 04:07 PM, Simon Loewenthal wrote:
>
>> Guess what? After removal of, local_phishing_reply.cf
99_anonwhois.cf malware.blocklist.cf the memory usage dropped to 15% of
RAM. Time to add more children into the mix. Cheers, S
>
> good to
hear...
> if not running already, Razor & Pyzor, even iXhash (tho
unsupported,
> still very effective) give you good results while
keeping cpu usage &
> memory lower than huge rule sets which hardly
hit.
Re: Upgrade from SA 3.3.1 to 3.3.2 - increase in memory requirements
on Debian 6
Posted by Axb <ax...@gmail.com>.
On 03/06/2013 04:07 PM, Simon Loewenthal wrote:
>
>
> Guess what? After removal of,
>
> local_phishing_reply.cf
>
>
> 99_anonwhois.cf
>
> malware.blocklist.cf
>
> the memory usage dropped to
> 15% of RAM.
>
> Time to add more children into the mix.
>
> Cheers, S
good to hear...
if not running already, Razor & Pyzor, even iXhash (tho unsupported,
still very effective) give you good results while keeping cpu usage &
memory lower than huge rule sets which hardly hit.
Re: Upgrade from SA 3.3.1 to 3.3.2 - increase in memory requirements on Debian 6
Posted by Simon Loewenthal <si...@klunky.co.uk>.
Guess what? After removal of,
local_phishing_reply.cf
99_anonwhois.cf
malware.blocklist.cf
the memory usage dropped to
15% of RAM.
Time to add more children into the mix.
Cheers, S
On
2013-03-06 15:55, Kevin A. McGrail wrote:
> On 3/6/2013 9:53 AM, Simon
Loewenthal wrote:
>
>> Hi KAM and AxB, The system is a small low cost
VM. The provider (for some reason) only offers to move the server to a
new box, instead of adding an extra half gig, which is pretty poor. I
don't have the time to spare for such a move for the moment. Yep - It's
64bit : amd64. Rule sets. I shall drop some rule sets. An sa-compile is
run every time the automatically downloaded rulesets change, but this
won't necessarily cut here when so tight on ram.
>
> Understood.
Definitely look at rule sets and using 64Bit for a machine
> with 500MB
of RAM just seems like a massive way to waste memory. When
> you move,
if you are locked to ~4GB or less, go with x86.
>
> Regards,
> KAM
Re: Upgrade from SA 3.3.1 to 3.3.2 - increase in memory requirements
on Debian 6
Posted by "Kevin A. McGrail" <KM...@PCCC.com>.
On 3/6/2013 9:53 AM, Simon Loewenthal wrote:
>
> Hi KAM and AxB,
>
> The system is a small low cost VM. The provider (for some reason)
> only offers to move the server to a new box, instead of adding an
> extra half gig, which is pretty poor. I don't have the time to spare
> for such a move for the moment. Yep - It's 64bit : amd64.
>
> Rule sets. I shall drop some rule sets. An sa-compile is run every
> time the automatically downloaded rulesets change, but this won't
> necessarily cut here when so tight on ram.
>
Undestood. Definitely look at rule sets and using 64Bit for a machine
with 500MB of RAM just seems like a massive way to waste memory. When
you move, if you are locked to ~4GB or less, go with x86.
Regards,
KAM
Re: Upgrade from SA 3.3.1 to 3.3.2 - increase in memory requirements
on Debian 6
Posted by Axb <ax...@gmail.com>.
I'll lower my price:
3 vouchers for medium BK menus :)
ok - enuff of OT :)
On 03/06/2013 05:10 PM, JK4 wrote:
>
>
> Hmm, we have some BL460 G1s we're gonna throw out soon. Hopefully
> some BL860 Itaniums as well next year.
>
> You can have those as well :)
>
>
> On 2013-03-06 17:04, Axb wrote:
>
>> On 03/06/2013 04:51 PM, Martin
> Gregorie wrote:
>>
>>> On Wed, 2013-03-06 at 15:53 +0100, Simon
> Loewenthal wrote:
>>>
>>>> Hi KAM and AxB, The system is a small low
> cost VM. The provider (for some reason) only offers to move the server
> to a new box, instead of adding an extra half gig, which is pretty poor.
> I don't have the time to spare for such a move for the moment. Yep -
> It's 64bit : amd64.
>>> Just a thought: It would be interesting to see,
> for a lowish mail volume, how well spamd would run on a headless
> RaspberryPi Model B. Its hard to think of a cheaper bit of kit to use.
> The current model Bs have 512MB of RAM, and work well when run headless.
> There are instructions available that detail how to configure a new
> RPi/B's SD card for headless operation before you boot it for the first
> time. Perl 5.14 is included in the standard RPi Debian distro. Up to the
> start of last year I was managing perfectly well with spamd on a 512Mb
> 866MHz P3 box running Fedora, where it got to share memory with Apache,
> Postgres, Postfix, getmail and Dovecot plus a fair development load, so
> I reckon that it should be fine on a 1GHz Rpi with the same sized
> memory.
>>
>> for the cost of a RPi I'll give you a HP DL360 G4 with 2 x
> 146GB SCSI
>> drives, 4 GB Ram. I'll even give you a spare drive. You'll
> have to pay
>> for freight from europe
>
>
Re: Upgrade from SA 3.3.1 to 3.3.2 - increase in memory requirements on Debian 6
Posted by JK4 <ju...@klunky.co.uk>.
Hmm, we have some BL460 G1s we're gonna throw out soon. Hopefully
some BL860 Itaniums as well next year.
You can have those as well :)
On 2013-03-06 17:04, Axb wrote:
> On 03/06/2013 04:51 PM, Martin
Gregorie wrote:
>
>> On Wed, 2013-03-06 at 15:53 +0100, Simon
Loewenthal wrote:
>>
>>> Hi KAM and AxB, The system is a small low
cost VM. The provider (for some reason) only offers to move the server
to a new box, instead of adding an extra half gig, which is pretty poor.
I don't have the time to spare for such a move for the moment. Yep -
It's 64bit : amd64.
>> Just a thought: It would be interesting to see,
for a lowish mail volume, how well spamd would run on a headless
RaspberryPi Model B. Its hard to think of a cheaper bit of kit to use.
The current model Bs have 512MB of RAM, and work well when run headless.
There are instructions available that detail how to configure a new
RPi/B's SD card for headless operation before you boot it for the first
time. Perl 5.14 is included in the standard RPi Debian distro. Up to the
start of last year I was managing perfectly well with spamd on a 512Mb
866MHz P3 box running Fedora, where it got to share memory with Apache,
Postgres, Postfix, getmail and Dovecot plus a fair development load, so
I reckon that it should be fine on a 1GHz Rpi with the same sized
memory.
>
> for the cost of a RPi I'll give you a HP DL360 G4 with 2 x
146GB SCSI
> drives, 4 GB Ram. I'll even give you a spare drive. You'll
have to pay
> for freight from europe
Re: Upgrade from SA 3.3.1 to 3.3.2 - increase in memory requirements
on Debian 6
Posted by Axb <ax...@gmail.com>.
On 03/06/2013 04:51 PM, Martin Gregorie wrote:
> On Wed, 2013-03-06 at 15:53 +0100, Simon Loewenthal wrote:
>>
>> Hi KAM and AxB,
>>
>> The system is a small low cost VM. The provider
>> (for some reason) only offers to move the server to a new box, instead
>> of adding an extra half gig, which is pretty poor. I don't have the time
>> to spare for such a move for the moment. Yep - It's 64bit : amd64.
>>
> Just a thought:
>
> It would be interesting to see, for a lowish mail volume, how well
> spamd would run on a headless RaspberryPi Model B. Its hard to think of
> a cheaper bit of kit to use. The current model Bs have 512MB of RAM, and
> work well when run headless. There are instructions available that
> detail how to configure a new RPi/B's SD card for headless operation
> before you boot it for the first time. Perl 5.14 is included in the
> standard RPi Debian distro.
>
> Up to the start of last year I was managing perfectly well with spamd on
> a 512Mb 866MHz P3 box running Fedora, where it got to share memory with
> Apache, Postgres, Postfix, getmail and Dovecot plus a fair development
> load, so I reckon that it should be fine on a 1GHz Rpi with the same
> sized memory.
for the cost of a RPi I'll give you a HP DL360 G4 with 2 x 146GB SCSI
drives, 4 GB Ram. I'll even give you a spare drive. You'll have to pay
for freight from europe
Re: Upgrade from SA 3.3.1 to 3.3.2 - increase in memory
requirements on Debian 6
Posted by Martin Gregorie <ma...@gregorie.org>.
On Wed, 2013-03-06 at 15:53 +0100, Simon Loewenthal wrote:
>
> Hi KAM and AxB,
>
> The system is a small low cost VM. The provider
> (for some reason) only offers to move the server to a new box, instead
> of adding an extra half gig, which is pretty poor. I don't have the time
> to spare for such a move for the moment. Yep - It's 64bit : amd64.
>
Just a thought:
It would be interesting to see, for a lowish mail volume, how well
spamd would run on a headless RaspberryPi Model B. Its hard to think of
a cheaper bit of kit to use. The current model Bs have 512MB of RAM, and
work well when run headless. There are instructions available that
detail how to configure a new RPi/B's SD card for headless operation
before you boot it for the first time. Perl 5.14 is included in the
standard RPi Debian distro.
Up to the start of last year I was managing perfectly well with spamd on
a 512Mb 866MHz P3 box running Fedora, where it got to share memory with
Apache, Postgres, Postfix, getmail and Dovecot plus a fair development
load, so I reckon that it should be fine on a 1GHz Rpi with the same
sized memory.
Martin
Re: Upgrade from SA 3.3.1 to 3.3.2 - increase in memory requirements on Debian 6
Posted by Simon Loewenthal <si...@klunky.co.uk>.
Hi KAM and AxB,
The system is a small low cost VM. The provider
(for some reason) only offers to move the server to a new box, instead
of adding an extra half gig, which is pretty poor. I don't have the time
to spare for such a move for the moment. Yep - It's 64bit : amd64.
Rule sets. I shall drop some rule sets. An sa-compile is run every
time the automatically downloaded rulesets change, but this won't
necessarily cut here when so tight on ram.
On 2013-03-06 15:36, Kevin
A. McGrail wrote:
> On 3/6/2013 9:17 AM, Simon Loewenthal wrote:
>
>> Options are : /usr/sbin/spamd --create-prefs -x -q --ipv4
--max-children 1 --timeout-child 180 --sql-config --nouser-config
--username spamd --helper-home-dir -s /var/log/spamd.log
--virtual-config-dir=/users/%d/%u -d --pidfile=/var/run/spamd.pid
>>
>> ( 1 child set because of lack of memory. 2 causes it to swap)
>>
>> As far as I can tell no rulesets have changed. I have these
additional ones added :
>>
>> # MALWARE BLOCKLIST
>>
"http://www.malwarepatrol.net/cgi/submit?action=list_sa" [1] -O
MALWARE.BLOCKLIST.CF
>>
>> # 99_ANONWHOIS
>>
"http://anonwhois.org/99_anonwhois.cf" [2] -O 99_ANONWHOIS.CF
>>
>> #
kAOS rESIgns MTX Blacklist
>> /usr/local/bin/mtx_blacklist.pl
>>
>> #
SOUGHT rules via sa-update
>> --channel SOUGHT.rules.yerp.org --channel
updates.spamassassin.org
>>
>> # Generate Spamassassin rules from the
phishing_reply_addresses list
>>
/usr/local/bin/addresses2spamassassin.pl - Prodcues file:
LOCAL_PHISHING_REPLY.CF
>>
>> CUSTOM RULES IN LOCAL.CF
>>
>> I have a
several custom rules comprising of 231 lines. These are simple rules
comprising of some simple regex. Some were copied from this mailing
list. I should turn these off and see what happens.
>>
>> In the past
24 hours the spamd memory usage has dropped to 198 Mb, which is a
relief, but this happened after I did a update on the server from
squeeze/updates, squeeze, and security. Before time I just had security
configured.
> Hi Simon,
>
> I've never really worked with a system that
tight on ram but I would definitely look at the configs you are adding.
Some of those look to change per day and the memory usage seems fairly
high.
>
> Here's an x86 system where I'm running a few spamds:
>
>
spamd 1088 46.5 2.1 98024 90144 ? R 08:31 29:40 spamd child
> root 14509
0.0 1.0 49964 44116 ? Ss Mar04 1:00 /usr/local/bin/spamd -d
--min-spare=1 --min-children=5 --max-spare=10 --max-conn-per-child=1000
--max-children=40 -q -x -u spamd --allowed-ips=127.0.0.1 -r
/var/run/spamd.pid
> spamd 14697 5.5 2.1 95192 87896 ? R Mar05 39:04
spamd child
> spamd 17369 3.4 1.6 73480 66964 ? S 00:18 19:04 spamd
child
> spamd 18328 1.9 2.2 99632 91804 ? S Mar04 40:01 spamd child
>
spamd 25112 1.0 2.2 100220 92116 ? S Mar05 19:55 spamd child
> spamd
28567 0.5 1.6 74424 67364 ? S Mar05 9:39 spamd child
> spamd 29384 0.0
1.1 54908 48348 ? R 03:52 0:05 spamd child
> spamd 29656 0.2 1.4 65020
58132 ? S Mar05 3:40 spamd child
> spamd 31115 0.0 1.0 49964 42432 ? S
05:01 0:00 spamd child
> spamd 32566 0.0 1.0 49964 42432 ? S 06:52 0:00
spamd child
>
> For comparison, your memory foot print seems higher.
Are you on a 64 bit system? Assuming note...
>
> regards,
> KAM
Links:
------
[1]
http://www.malwarepatrol.net/cgi/submit?action=list_sa
[2]
http://anonwhois.org/99_anonwhois.cf
Re: Upgrade from SA 3.3.1 to 3.3.2 - increase in memory requirements
on Debian 6
Posted by "Kevin A. McGrail" <KM...@PCCC.com>.
On 3/6/2013 9:17 AM, Simon Loewenthal wrote:
> Options are : /usr/sbin/spamd --create-prefs -x -q --ipv4
> --max-children 1 --timeout-child 180 --sql-config --nouser-config
> --username spamd --helper-home-dir -s /var/log/spamd.log
> --virtual-config-dir=/users/%d/%u -d --pidfile=/var/run/spamd.pid
>
> ( 1 child set because of lack of memory. 2 causes it to swap)
>
> As far as I can tell no rulesets have changed. I have these
> additional ones added :
>
>
> # MALWARE BLOCKLIST
> "http://www.malwarepatrol.net/cgi/submit?action=list_sa" -O
> *malware.blocklist.cf*
>
> # 99_ANONWHOIS
> "http://anonwhois.org/99_anonwhois.cf" -O *99_anonwhois.cf*
>
> # kAOS rESIgns MTX Blacklist
> /usr/local/bin/mtx_blacklist.pl
>
> # SOUGHT rules via sa-update
> --channel *sought*.rules.yerp.org --channel updates.spamassassin.org
>
>
> # Generate Spamassassin rules from the phishing_reply_addresses list
> /usr/local/bin/addresses2spamassassin.pl - Prodcues
> file:*local_phishing_reply.cf*
>
> *Custom rules in local.cf*
>
> I have a several custom rules comprising of 231 lines. These are
> simple rules comprising of some simple regex. Some were copied from
> this mailing list. I should turn these off and see what happens.
>
> In the past 24 hours the spamd memory usage has dropped to 198 Mb,
> which is a relief, but this happened after I did a update on the
> server from squeeze/updates, squeeze, and security. Before time I just
> had security configured.
>
Hi Simon,
I've never really worked with a system that tight on ram but I would
definitely look at the configs you are adding. Some of those look to
change per day and the memory usage seems fairly high.
Here's an x86 system where I'm running a few spamds:
spamd 1088 46.5 2.1 98024 90144 ? R 08:31 29:40 spamd child
root 14509 0.0 1.0 49964 44116 ? Ss Mar04 1:00
/usr/local/bin/spamd -d --min-spare=1 --min-children=5 --max-spare=10
--max-conn-per-child=1000 --max-children=40 -q -x -u spamd
--allowed-ips=127.0.0.1 -r /var/run/spamd.pid
spamd 14697 5.5 2.1 95192 87896 ? R Mar05 39:04 spamd child
spamd 17369 3.4 1.6 73480 66964 ? S 00:18 19:04 spamd child
spamd 18328 1.9 2.2 99632 91804 ? S Mar04 40:01 spamd child
spamd 25112 1.0 2.2 100220 92116 ? S Mar05 19:55 spamd child
spamd 28567 0.5 1.6 74424 67364 ? S Mar05 9:39 spamd child
spamd 29384 0.0 1.1 54908 48348 ? R 03:52 0:05 spamd child
spamd 29656 0.2 1.4 65020 58132 ? S Mar05 3:40 spamd child
spamd 31115 0.0 1.0 49964 42432 ? S 05:01 0:00 spamd child
spamd 32566 0.0 1.0 49964 42432 ? S 06:52 0:00 spamd child
For comparison, your memory foot print seems higher. Are you on a 64
bit system? Assuming note...
regards,
KAM