You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by David Kentwood <da...@gmail.com> on 2012/07/08 12:49:14 UTC
setup spamassassin without amavisd
Hi,
I want to setup spamassassin + clamav + postfix. Many internet guides use
Amavisd to integrate them together. However, my vps has only 516mb ram so I
don't want to install Amavisd unless it's really recommended. So would the
setup work well without using Amavisd? Would you recommend using Amavisd?
Thank you,
Dave
Re: setup spamassassin without amavisd
Posted by Frederic De Mees <li...@demees.net>.
From: "Jari Fredriksson" <ja...@iki.fi>
> On Sun, July 8, 2012 13:49, David Kentwood wrote:
>> Hi,
>>
>> I want to setup spamassassin + clamav + postfix. Many internet guides use
>> Amavisd to integrate them together. However, my vps has only 516mb ram so
>> I
>> don't want to install Amavisd unless it's really recommended. So would
>> the
>> setup work well without using Amavisd? Would you recommend using Amavisd?
>>
>> Thank you,
>>
>> Dave
>>
>
> I use it without amavisd, and works fine. I have different installations,
>
> - integrated via using postfix/spamass-milter
> - integrated via postfix/maildrop
>
> Those have differences, and I have different requiremens for each
> installation and those do it fine.
>
> However, amavisd. It links to SpamAssasin by using it as an add-on
> library. It does not need spamd up, amavisd is the daemon itself. I think
> SpamAssassin + amavisd memory requirement does not differ much from using
> plain SpamAssassin. I have not verified this, but this is my
> understanding.
I use a combination with Postfix and spampd as a policy daemon.
In this setup the policy daemon is called in real time during the SMTP
transaction, and the incoming mail message which exhibits a SA score above a
certain value is rejected at the end of DATA.
My installation runs without any incident since 2 or 3 years on boxes with
512M RAM. But the message volume is not high, no more than 10'000 ~ 20'000
messages a day, including rejected spam.
See this information regarding spampd:
http://wiki.apache.org/spamassassin/IntegratePostfixViaSpampd
I have posted a description of my setup, in French:
http://blog.demees.net/?p=7
Frédéric De Mees
Brussels
Re: setup spamassassin without amavisd
Posted by Jari Fredriksson <ja...@iki.fi>.
On Sun, July 8, 2012 13:49, David Kentwood wrote:
> Hi,
>
> I want to setup spamassassin + clamav + postfix. Many internet guides use
> Amavisd to integrate them together. However, my vps has only 516mb ram so
> I
> don't want to install Amavisd unless it's really recommended. So would the
> setup work well without using Amavisd? Would you recommend using Amavisd?
>
> Thank you,
>
> Dave
>
I use it without amavisd, and works fine. I have different installations,
- integrated via using postfix/spamass-milter
- integrated via postfix/maildrop
Those have differences, and I have different requiremens for each
installation and those do it fine.
However, amavisd. It links to SpamAssasin by using it as an add-on
library. It does not need spamd up, amavisd is the daemon itself. I think
SpamAssassin + amavisd memory requirement does not differ much from using
plain SpamAssassin. I have not verified this, but this is my
understanding.
Re: setup spamassassin without amavisd
Posted by Per Jessen <pe...@computer.org>.
David Kentwood wrote:
> Hi,
>
> I want to setup spamassassin + clamav + postfix. Many internet guides
> use Amavisd to integrate them together. However, my vps has only 516mb
> ram so I don't want to install Amavisd unless it's really recommended.
> So would the setup work well without using Amavisd?
Yes, it works very well without amavisd:
http://jessen.ch/articles/spamassassin-and-postfix/
(a bit old, but still valid).
--
Per Jessen, Zürich (20.5°C)
Re: setup spamassassin without amavisd
Posted by Martin Gregorie <ma...@gregorie.org>.
On Sun, 2012-07-08 at 06:49 -0400, David Kentwood wrote:
> Hi,
>
> I want to setup spamassassin + clamav + postfix. Many internet guides use
> Amavisd to integrate them together. However, my vps has only 516mb ram
>
You don't say what your mail volume is, but until recently I ran SA
successfully on a 512MB, 866 MHz P3 house server box which also runs
Postgres and Apache. Mail volume was 150 msgs/day +/-50. I'm still using
the same mail handling chain but for reasons not related to mail, the
box recently got bigger and faster by a factor of 8 for RAM and 12 for
CPU cycles.
The set up is that SA only handles incoming mail: getmail fetches the
incoming mail from my ISP and the getmail mda script defines a pipeline
consisting of:
spamc | spamkiller | sendmail
where (obviously) spamc passes messages to spamd for processing.
Spamkiller is a locally developed program that looks at the SA headers,
quarantines spam and hands ham to the Postfix sendmail utility for
delivery the local copy of Postfix. If this type of set-up floats your
boat, you can download spamkiller and friends from here:
http://www.libelle-systems.com/free/
HTH
Martin
Re: setup spamassassin without amavisd
Posted by Benny Pedersen <me...@junc.org>.
Den 2012-07-08 12:49, David Kentwood skrev:
> I want to setup spamassassin + clamav + postfix.
good choice
> Many internet guides use Amavisd to integrate them together.
this is the most common setup, it does not mean there is not any other
options
> However, my vps has only 516mb
i have being there with freebsd 4.9 with just 256mb
> ram so I dont want to install Amavisd unless its really recommended.
amavisd itself is not ram hungry, most ram usa is from clamd and
spamassasin perl usages
> So would the setup work well without using Amavisd?
i would start seeing what is left after clamd is started
> Would you recommend using Amavisd?
i would upgrade to 1024mb
and try using spampd and clamav-milter, spampd can be used as proxy in
postfix so you dont need an quarantine if harddisk space is small
but there is one caviat, proxy scanning limits speeds in postfix since
it queue up before in queue, later versions of postfix have better
support for proxy scanning
Re: setup spamassassin without amavisd
Posted by David Kentwood <da...@gmail.com>.
Thank you all for your replies. I have carefully considered all of your
suggestions and decided to try the Amavisd setup. According to some
suggestions here and various online sources, Amavisd is fast, scalable,
easy to use and highly configurable. It loads up without spamd, and can
handles spam and virus filtering simultaneously.
I did the setup on a centos 6.2 vps running nginx/php/mysql/bind together
with postfix/dovecot/clamav/spamassassin/amavisd. I didn't use any custom
rule set for clamav/spamassassin. Most of the settings were based on
default install using yum. However, after starting all the services, the
entire memory usage shot up to around 900mb (the openvz vps only has 512mb
ram + 512mb burst memory) while the server was not even live!
free -k
total used free shared
buffers cached
Mem: 1048576 894248 154328 0 0 0
-/+ buffers/cache: 894248 154328
Swap: 0 0 0
Moreover, the 'top' command showed that Amavisd and clamd each reserved
about 300mb of virtual memory. I am not sure if this is typical for such a
setup. Perhaps high memory allocation is caused by some intrinsic issue
with the openvz kernel?! I'll be doing some customization per suggestions
from some of the replies here. And if there aren't ways to significally
reduce the memory usage, I might just upgrade the memory to 1gb.
Regardless, thank you very much for all your help!
Re: Perl, fork, and copy-on-write (was Re: setup spamassassin
without amavisd)
Posted by Henrik K <he...@hege.li>.
On Mon, Jul 09, 2012 at 04:06:48AM -0400, David F. Skoll wrote:
> On Mon, 9 Jul 2012 09:06:39 +0300
> Henrik K <he...@hege.li> wrote:
>
> > You can easily run many children since amavisd or spamd forks are
> > copy-on-writed pretty well. So only extra memory used is the per
> > scan state and file data etc.
>
> Have you actually measured this? My experience is that forked Perl
> processes end up sharing surprisingly little memory. The reason is
> that Perl uses reference-counting to track liveness (so it can free
> memory when the reference count reaches zero.) This turns a lot of
> what should be read-only accesses into writes and severely hurts
> memory page sharing.
Yes I tested it quite a bit with amavis, sadly I don't have my notes
anymore. I think each active child added around 5-10MB to used system
memory seen with top/free commands.
Since childs can be configured to be shut down after x requests, it also
reduces any "risks". Of course YMMV and there could be flaws in my testing.
But if 20 childs happily start and run on my small VPS, it should already
tell something. ;-)
Perl, fork, and copy-on-write (was Re: setup spamassassin without
amavisd)
Posted by "David F. Skoll" <df...@roaringpenguin.com>.
On Mon, 9 Jul 2012 09:06:39 +0300
Henrik K <he...@hege.li> wrote:
> You can easily run many children since amavisd or spamd forks are
> copy-on-writed pretty well. So only extra memory used is the per
> scan state and file data etc.
Have you actually measured this? My experience is that forked Perl
processes end up sharing surprisingly little memory. The reason is
that Perl uses reference-counting to track liveness (so it can free
memory when the reference count reaches zero.) This turns a lot of
what should be read-only accesses into writes and severely hurts
memory page sharing.
Be careful.
Regards,
David.
Re: setup spamassassin without amavisd
Posted by Henrik K <he...@hege.li>.
On Mon, Jul 09, 2012 at 11:29:08AM -0400, Bowie Bailey wrote:
> On 7/9/2012 5:42 AM, Benny Pedersen wrote:
> > Den 2012-07-09 08:06, Henrik K skrev:
> >
> >> I just ditch main.cld which seems pointless, I think it saved
> >> something like
> >> 40-50MB. If there are actually ever any new "viruses", daily.cld
> >> should
> >> catch them. With this and most 3rd party sigs, clamd is only 80MB
> >> RSS.
> > so you think that virus that are very old are moved from main into
> > daily if seen daily ?
> >
> > if thats the case i can save some mem aswell :=)
> >
> > freshclam will just try to keep atleast main and daily :(
>
> I'm not sure this is safe. The viruses in main.cld may be older than
> the ones in daily.cld, but that doesn't mean they aren't still out
> there. In answer to your question, I would very much doubt that virus
> definitions are every moved from main back to daily. Since you are
> expected to have the main.cld database, which already contains the
> definition, it would be a waste of bandwidth to force everyone to
> download it again in daily.cld.
>
> You should ask on the ClamAV list for a definitive answer.
It's been months since last main generation.
30750647 Mar 23 13:15 main.cvd
Since I couldn't care less about any "viruses" which are pretty much
non-existent today, I have no problem of doing this. I mainly use ClamAV
for all the 3rd party sigs. MTA rules and SpamAssassin catches anything
resembling a virus here anyway, but keeping daily sigs is fine as backup.
And yes you need a custom download directory for freshclam to keep all the
sigs and move daily.cld manually to the "real" directory.
Re: setup spamassassin without amavisd
Posted by Bowie Bailey <Bo...@BUC.com>.
On 7/9/2012 5:42 AM, Benny Pedersen wrote:
> Den 2012-07-09 08:06, Henrik K skrev:
>
>> I just ditch main.cld which seems pointless, I think it saved
>> something like
>> 40-50MB. If there are actually ever any new "viruses", daily.cld
>> should
>> catch them. With this and most 3rd party sigs, clamd is only 80MB
>> RSS.
> so you think that virus that are very old are moved from main into
> daily if seen daily ?
>
> if thats the case i can save some mem aswell :=)
>
> freshclam will just try to keep atleast main and daily :(
I'm not sure this is safe. The viruses in main.cld may be older than
the ones in daily.cld, but that doesn't mean they aren't still out
there. In answer to your question, I would very much doubt that virus
definitions are every moved from main back to daily. Since you are
expected to have the main.cld database, which already contains the
definition, it would be a waste of bandwidth to force everyone to
download it again in daily.cld.
You should ask on the ClamAV list for a definitive answer.
--
Bowie
Re: setup spamassassin without amavisd
Posted by Benny Pedersen <me...@junc.org>.
Den 2012-07-09 08:06, Henrik K skrev:
> I just ditch main.cld which seems pointless, I think it saved
> something like
> 40-50MB. If there are actually ever any new "viruses", daily.cld
> should
> catch them. With this and most 3rd party sigs, clamd is only 80MB
> RSS.
so you think that virus that are very old are moved from main into
daily if seen daily ?
if thats the case i can save some mem aswell :=)
freshclam will just try to keep atleast main and daily :(
Re: setup spamassassin without amavisd
Posted by Henrik K <he...@hege.li>.
On Sun, Jul 08, 2012 at 04:40:31PM -0500, Dave Funk wrote:
> >On 07/08/2012 12:49 PM, David Kentwood wrote:
> >>Hi,
> >>
> >>I want to setup spamassassin + clamav + postfix. Many internet guides use
> >>Amavisd to integrate them together. However, my vps has only 516mb ram so I
> >>don't want to install Amavisd unless it's really recommended. So would the
> >>setup work well without using Amavisd? Would you recommend using Amavisd?
> >>
>
> One thing to keep in mind are the various factors that influence
> memory usage in spamassassin & clamav (and by how much).
>
> For example (on a SLES-11 x86_64 box) clamd with just the stock ClamAV
> rules has a RSS of 155MB, with a number of 3'rd party add in rulesets
> (EG Sanesecurity, SecureiteInfo, etc) its RSS is over 500MB.
> However the Clam + added rulesets has a hit rate that is 50x~100x higher
> than just stock ClamAv rules
I just ditch main.cld which seems pointless, I think it saved something like
40-50MB. If there are actually ever any new "viruses", daily.cld should
catch them. With this and most 3rd party sigs, clamd is only 80MB RSS.
> spamd's memory size is influenced by added rules and by scanned
> message size. As spamd keeps in memory multiple copies of a message
> (the raw form, the parsed 'full' form, the "cleaned" normalized
> form, etc) its memory
> usage grows nonlinearly with message size. EG if you restrict spamd
> to only scanning small (< 64KB) messages it might be no more than
> 100MB RSS but when you feed it larger messages (say 350KB) it can
> easily hit 150MB RSS per instance.
I've never seen my amavisd RSS over 100MB (512k msg size). On a 64-bit box
it can be something like 1.5x more since Perl likes to spend a whole lot
more memory there. But not something to worry usually on a VPS.
> So if you limit scanned message size you use less memory but then bloated
> spams will slip thru.
They won't if you use amavisd. It just truncates messages to the limit and
scans that. :-)
> Depending upon your mail flow rate you may want to keep multiple
> spamd children around. Each child uses up memory but multiple
> children help thruput during bursts of incoming messages.
You can easily run many children since amavisd or spamd forks are
copy-on-writed pretty well. So only extra memory used is the per scan state
and file data etc.
Re: setup spamassassin without amavisd
Posted by Dave Funk <db...@engineering.uiowa.edu>.
> On 07/08/2012 12:49 PM, David Kentwood wrote:
>> Hi,
>>
>> I want to setup spamassassin + clamav + postfix. Many internet guides use
>> Amavisd to integrate them together. However, my vps has only 516mb ram so I
>> don't want to install Amavisd unless it's really recommended. So would the
>> setup work well without using Amavisd? Would you recommend using Amavisd?
>>
One thing to keep in mind are the various factors that influence memory
usage in spamassassin & clamav (and by how much).
For example (on a SLES-11 x86_64 box) clamd with just the stock ClamAV
rules has a RSS of 155MB, with a number of 3'rd party add in rulesets
(EG Sanesecurity, SecureiteInfo, etc) its RSS is over 500MB.
However the Clam + added rulesets has a hit rate that is 50x~100x higher
than just stock ClamAv rules
spamd's memory size is influenced by added rules and by scanned message
size. As spamd keeps in memory multiple copies of a message (the raw form,
the parsed 'full' form, the "cleaned" normalized form, etc) its memory
usage grows nonlinearly with message size. EG if you restrict spamd to
only scanning small (< 64KB) messages it might be no more than 100MB RSS
but when you feed it larger messages (say 350KB) it can easily hit
150MB RSS per instance.
So if you limit scanned message size you use less memory but then
bloated spams will slip thru. Also 3'rd party rulesets can be
quite helpful hitting fast mutating spams.
Depending upon your mail flow rate you may want to keep multiple spamd
children around. Each child uses up memory but multiple children help
thruput during bursts of incoming messages.
--
Dave Funk University of Iowa
<dbfunk (at) engineering.uiowa.edu> College of Engineering
319/335-5751 FAX: 319/384-0549 1256 Seamans Center
Sys_admin/Postmaster/cell_admin Iowa City, IA 52242-1527
#include <std_disclaimer.h>
Better is not better, 'standard' is better. B{
Re: setup spamassassin without amavisd
Posted by Axb <ax...@gmail.com>.
On 07/08/2012 12:49 PM, David Kentwood wrote:
> Hi,
>
> I want to setup spamassassin + clamav + postfix. Many internet guides use
> Amavisd to integrate them together. However, my vps has only 516mb ram so I
> don't want to install Amavisd unless it's really recommended. So would the
> setup work well without using Amavisd? Would you recommend using Amavisd?
>
First, you'd need to check if you need any features which only Amavisd
can offer.
If the answer is "no", I'd go for Fuglu.
It uses spamd, interfaces with Clamav, etc.
Written in Python, real easy to setup & manage.
Well documented and under active development.
Been using in production for quite a while, on a dozen of high traffic
boxes and happy :)
Docs:
http://sourceforge.net/apps/trac/fuglu/
Download:
http://sourceforge.net/projects/fuglu/
Axb