Postfix, ClamAV, and SA

[Chris <ra...@makeworld.com>]

Can a mail server just be Postfix, ClamAV, and SA without the need for
Mailscanner or Amavis?

If so - I would like to see a how-to is someone has one.


-- 
Best regards,
Chris

Most people want to be delivered from temptation but
would like it to keep in touch.

Re: Postfix, ClamAV, and SA

[Mark Martinec <Ma...@ijs.si>]

From SickBoy,
> It is true, that amavisd-new preloads some of the SA code, but it is
> also true, that this invocation of SA is more like using the spamassasin
> commandline, meaning very,very resource-hungy.

This is a gross misinformation. amavisd-new behaves pretty much
as spamd does, the main difference is that it uses smtp protocol
instead of pipe+spamc to talk with MTA. Performance-wise both the
amavisd-new and the spamc+spamd are in the same category,
and like any other tool, both can be mistuned for a particular environment,
and are subject to the same SA setup/tuning/bottleneck-avoidance guidelines.

> After carefull studyin this doc :
> http://svn.apache.org/repos/asf/spamassassin/branches/3.1/spamd/README ,
> I've  realised that's just the thing I need. Running SA as daemon, like
> clamd.

amavisd is a daemon and it behaves much like spamd.

> Unfortunately that's where the problem started to emerge.
> Amavisd-new can NOT use daemonized version of SA, meaning invoking the
> scaning thru spamc/spamd pair like it does for ClamAV.

Misinformation.

  Mark

Re: Postfix, ClamAV, and SA

[mouss <us...@free.fr>]

Chris a écrit :
> While I agree with most of what you outlined (I too experiance tons of
> issues with the nes SA/Amavisd combo) - My mailsystem won't work (at
> least I don't think so) under this config.

unless you can measure and check what causes the real problems, ther's
no point in trying to devise a theory on that. trading a perl based
daemon (amavisd-new) for a shell script that is forked not only for
every message, but for every recipient can hardly be justified. If the
previous message was sent to the squid guys, I don't promiss they'll
keep the sender alife...

> 
> Mainly - it's a mailsever that is a relay for an exchange server. That
> means, there isnt any local users. so to me the injection of procmail
> etc, would be needlessly wasting time to implement.
> 
> Even with a gig a ram, I need to trim back the amount of rules I use. In
> addition, I alwasy though the idea of a second smtp engine (as in
> Amavis) is silly.
> 
> I considered Mailscanner, I like the idea of NOT introducing another
> smtp engine however, I don't like the idea of HOLDing the mail either.
> 

The issue isn't the number of tcp connections. It's about what each
does. The real issue when using SA is that you are doing a lot of regex
matching. and this will still be here, whether you write this in a C
daemon or in script.

> And as of late, Amaivs has been more like deailing with a 4 year old in
> a toy store.
> 
> So for now, I'm still looking and open to ideas.
> 

use another filter? I personally focus on getting less FPs while
catching most spam, than trying to get my mail faster.

Re: Postfix, ClamAV, and SA

[tuxstrike <tu...@gmail.com>]

Do you know MAIA MAILGUARD ? 

try it, http://www.maiamailguard.com/

i think that is better Mailscanner...

Em Dom, 2005-12-18 às 20:31 +0100, Kai Schaetzl escreveu:
> Chris wrote on Sun, 18 Dec 2005 10:44:47 -0600:
> 
> > I considered Mailscanner
> 
> Good step. Lots of options, nice web interface and bulk scanning of 
> messages.
> 
> Kai
> 

Re: Postfix, ClamAV, and SA

[Kai Schaetzl <ma...@conactive.com>]

Chris wrote on Sun, 18 Dec 2005 10:44:47 -0600:

> I considered Mailscanner

Good step. Lots of options, nice web interface and bulk scanning of 
messages.

Kai

-- 
Kai Schätzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com

Re: Postfix, ClamAV, and SA

[Chris <ra...@makeworld.com>]

SickBoy wrote:
> mouss wrote:
> 
>>Chris a écrit :
>>
>>
>>>Can a mail server just be Postfix, ClamAV, and SA without the need for
>>>Mailscanner or Amavis?
>>>
>>>If so - I would like to see a how-to is someone has one.
>>>
>>>
>>
>>yes, but why? you can use clamsmtp[d] for clamav. but if you're using
>>SA, then amavisd-new is recommended. It will run SA code directly,
>>without forking a spamassassin/spamc for every mail.
>>
>>there's not much justification for removing amavisd-new unless you also
>>want to remove SA.
>>
>>if you want per-user SA, then just run SA from an MDA (maildrop,
>>procmail, ...).
>>
> 
> 
> Howdy Gents ;)
> 
> Unfortunatelly for Mouss I have to disagree.
> I was a "happy" user of postfix+amavisd-new+sa.
> My system runs on dual p3@500 with 256 megs o' ram smoothly powered by
> carefully crafted gentoo ..
> 
> This combination worked fine for a couple (30 or so) users inside my net.
> 
> The problem started to emerge when me, and a bunch of my fellow
> net-citizen set up fetchmail to d/l all the msgs from different mail
> accounts we have.
> 
> The system started to choke seriosuly , consuming lot of memory, making
> swap to "grow" bigger and bigger.
> 
> It is true, that amavisd-new preloads some of the SA code, but it is
> also true, that this invocation of SA is more like using the spamassasin
> commandline, meaning very,very resource-hungy.
> 
> After carefull studyin this doc :
> http://svn.apache.org/repos/asf/spamassassin/branches/3.1/spamd/README ,
> I've  realised that's just the thing I need. Running SA as daemon, like
> clamd.

While I agree with most of what you outlined (I too experiance tons of
issues with the nes SA/Amavisd combo) - My mailsystem won't work (at
least I don't think so) under this config.

Mainly - it's a mailsever that is a relay for an exchange server. That
means, there isnt any local users. so to me the injection of procmail
etc, would be needlessly wasting time to implement.

Even with a gig a ram, I need to trim back the amount of rules I use. In
addition, I alwasy though the idea of a second smtp engine (as in
Amavis) is silly.

I considered Mailscanner, I like the idea of NOT introducing another
smtp engine however, I don't like the idea of HOLDing the mail either.

And as of late, Amaivs has been more like deailing with a 4 year old in
a toy store.

So for now, I'm still looking and open to ideas.




-- 
Best regards,
Chris

Consultants are mystical people who ask a company for
a number and then give it back to them.

Re: Postfix, ClamAV, and SA

[SickBoy <si...@blupill.com>]

mouss wrote:
> Chris a écrit :
> 
>> Can a mail server just be Postfix, ClamAV, and SA without the need for
>> Mailscanner or Amavis?
>>
>> If so - I would like to see a how-to is someone has one.
>>
>>
> 
> yes, but why? you can use clamsmtp[d] for clamav. but if you're using
> SA, then amavisd-new is recommended. It will run SA code directly,
> without forking a spamassassin/spamc for every mail.
> 
> there's not much justification for removing amavisd-new unless you also
> want to remove SA.
> 
> if you want per-user SA, then just run SA from an MDA (maildrop,
> procmail, ...).
> 

Howdy Gents ;)

Unfortunatelly for Mouss I have to disagree.
I was a "happy" user of postfix+amavisd-new+sa.
My system runs on dual p3@500 with 256 megs o' ram smoothly powered by
carefully crafted gentoo ..

This combination worked fine for a couple (30 or so) users inside my net.

The problem started to emerge when me, and a bunch of my fellow
net-citizen set up fetchmail to d/l all the msgs from different mail
accounts we have.

The system started to choke seriosuly , consuming lot of memory, making
swap to "grow" bigger and bigger.

It is true, that amavisd-new preloads some of the SA code, but it is
also true, that this invocation of SA is more like using the spamassasin
commandline, meaning very,very resource-hungy.

After carefull studyin this doc :
http://svn.apache.org/repos/asf/spamassassin/branches/3.1/spamd/README ,
I've  realised that's just the thing I need. Running SA as daemon, like
clamd.

Unfortunately that's where the problem started to emerge.
Amavisd-new can NOT use daemonized version of SA, meaning invoking the
scaning thru spamc/spamd pair like it does for ClamAV.

I also wasn't totally pleased with the "way" of mail before gettin' into
my mailbox, I mean POSTFIX by it's nature has no easy way for setting up
a pre-cleanup (as it happens in sendmail, where all the scanning is done
by milters BEFORE the mail actually gets into the queue), but it
redirects the mail to the next sptmd process spawned by postfix (port
10025), where the actual amavisd-new listens for conn and do the
scanning, and sends it back to localhost with headers rewritten (or
sends the info bout the virus/spam, according to the amavis conf.

So, the mail passes thru 2 more "hops" after entering a system, which
not only slows downe the process a bit, but also adds unnecessary
headers to the mail (2x recieved from localhost) which of course may be
a cosmetic issue, but I wanted the sys to behave exactly as I planned
(isn't that what's the linux for ? ;)

After stubling in the darkness a bit, tryin to fogure out what
combination of programs will work for me, I've anded up wit a solution.

There is a small script-based proggie called clamassasin, which uses a
daemonized version of clamav for mail scanning, and also rewrites the
msg headers, adding X-Virus* heads according to the state of mail that
just've been scanned.

After a bunch of changes to the script, to suit my needs and please my
eye (hehe) it finally fitted in place.

So now, the entire mailsys look like this:

ClamAV and SA running as daemons (runs as one dedicated user)

both scannings (SA and ClamAV) are invoked from procmailrc

postfix -> procmail (with site-wide conf, but every user can actually do
his own sorting of the mail according to the information in rewritten
subjects and/or X-* headers (this is like default_pass for both spam and
virus mails in amavis), beacuse all of my users are bright enough NOT to
be accidentaly infected by the virus that just came in, also because I'm
too busy to write the script that will redirect the virusmail to
'virustrap' account or to quarantine folder and inform (or not) the
receipent (like amavisd-new does).

The result ? Wohoo, amazing for me ..

Not only the entire scan process is taking ~half the amavisd-new time,
eating half of the ram (comparing to amavisd-new+clamav(daemonized)+sa)
for freshly started processes, but more importatntly, wht it actually DO
the scannning it's not eatin (like amavis does) huge chunks of memory
every time the mail comes in (behaving to me more likely the memory-leak
issue, not the normal program operation)

Finally, the mail is  *more* cleaner, there is NO annoying additional
headers when the mail hits it's shelf (recieved from localhost ..) and
all the X-Scanned headers are both in place and order I wanted them to
be ... (yeah!)

I can only guess, that scalability of this system may be mainly affected
by the pipe throughput, coz all the scanned "material" is pumped thru
pipes to clamd and spamd, but so far (average 3,5k mails/day) it runs
smoothly as planned.

So ...

Of course I know that's only my approach, and more could disagree, but
.. after a big dissapointment with amavisd-new,this one suits me well,
and with a few easy tuneups it can also be forced to automatically learn
spam that slipped thru and deleting false-positives (simply 2 more
default maildirs, spam and nospam; a cron job to do the scanning of
these folders, et .. voilla) ..

If someone is really interested in this example, I think I could provide
 the more detailed path, in form of a small howto.

This is the beauty of *nix. U can do every job in a few different ways.



-- 
Regards,
SickBoy <sickboy at blupill dot com>

# Encrypted/Signed Mail Preferred.
# Public Key ID: 0x2C6FC5FE (Available on key servers)
# 61A4 EE81 CDA3 52AB 04F3 3F4A CB3B B6D5 2C6F C5FE
# BluPill Enterprises @ http://www.blupill.com

Re: Postfix, ClamAV, and SA

[mouss <us...@free.fr>]

Chris a écrit :
> Can a mail server just be Postfix, ClamAV, and SA without the need for
> Mailscanner or Amavis?
> 
> If so - I would like to see a how-to is someone has one.
> 
> 

yes, but why? you can use clamsmtp[d] for clamav. but if you're using 
SA, then amavisd-new is recommended. It will run SA code directly, 
without forking a spamassassin/spamc for every mail.

there's not much justification for removing amavisd-new unless you also 
want to remove SA.

if you want per-user SA, then just run SA from an MDA (maildrop, 
procmail, ...).