You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Steve Yuroff <sy...@hiebing.com> on 2004/03/09 22:41:48 UTC
How to optimize SA?
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
I've been running SA on a Cobalt Qube since 2.42 was current, and it's
been a wonderful solution. Recently I moved to running spamd -d to
because of the increased load the increase in spam was putting on the
server. Now that's not cutting it- they're hitting me so hard I'm
seeing processor loads in the 30's, and my poor little Qube is buckling
under the load- I can't ssh in, email exchanges time out, ect.
What else can I do to make SA run as quickly as possible- besides the
obvious "buy a bigger server"?
Thanks,
Steve.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (Darwin)
iD8DBQFATjmxw0v3pepO1lMRAq8TAJ0VggCnqdAcyxyiJMoZaWKr4z+tSwCgiMZG
1ecxpcyXJnBqtJ96TTgXghU=
=/KqD
-----END PGP SIGNATURE-----
Re: How to optimize SA?
Posted by Steve Yuroff <sy...@hiebing.com>.
On Mar 9, 2004, at 3:46 PM, Jason Borkowsky wrote:
>
>> I've been running SA on a Cobalt Qube since 2.42 was current, and it's
>> been a wonderful solution. Recently I moved to running spamd -d to
>> because of the increased load the increase in spam was putting on the
>> server. Now that's not cutting it- they're hitting me so hard I'm
>> seeing processor loads in the 30's, and my poor little Qube is
>> buckling
>> under the load- I can't ssh in, email exchanges time out, ect.
>>
>> What else can I do to make SA run as quickly as possible- besides the
>> obvious "buy a bigger server"?
>
> I don't think tuning is going to get your loads down much from the
> 30's.
> Assuming a best case optimization gives you a 10% to 15% improvement,
> your
> box will still be overloaded. How many emails per hour/day are you
> receiving
> and over what link speed?
>
>
Last 24 hr period (4A-4A) was 1350 messages, over a 768K DSL line.
Re: How to optimize SA?
Posted by Jason Borkowsky <jc...@tcpns.com>.
> I've been running SA on a Cobalt Qube since 2.42 was current, and it's
> been a wonderful solution. Recently I moved to running spamd -d to
> because of the increased load the increase in spam was putting on the
> server. Now that's not cutting it- they're hitting me so hard I'm
> seeing processor loads in the 30's, and my poor little Qube is buckling
> under the load- I can't ssh in, email exchanges time out, ect.
>
> What else can I do to make SA run as quickly as possible- besides the
> obvious "buy a bigger server"?
I don't think tuning is going to get your loads down much from the 30's.
Assuming a best case optimization gives you a 10% to 15% improvement, your
box will still be overloaded. How many emails per hour/day are you receiving
and over what link speed?
Re: How to optimize SA?
Posted by Daniel Quinlan <qu...@pathname.com>.
Steve Yuroff <sy...@hiebing.com> writes:
> That's the first report I've had of the spamlist.org database bouncing
> mail from a legit source.
Well, they blocked my entire /24 network at PacBell. These addresses
are quite static and I'm not aware of any spammers being on it.
> Are you saying that spamlist.org and the add-in rulesets I'm using are
> crap? They seemed to be useful, legit tools to me, but I'm here to
> learn. If it's to be avoided, I'm all for it.
I'm saying if you have a performance problem, start with the default
installation and see if it has a performance problem before you start
trying to optimize things.
> My understanding was that the spamlist.org list would cut my processing
> load at the MTA level... isn't it easier to not accept the mail, than
> to accept it and then process it for ham/spam?
Yes and no.
First, it's very easy to reject messages. You can get no spam (and not
necessarily many complaints since nobody will be able to mail you) if
you bounce most everything.
Second, most MTAs probably don't do as good a job handling DNS
blacklists that are down, so it could be actually slower. It's
definitely not as accurate as I discovered.
Let me be blunt, when I reply to a question on a mailing list and I get
a bounce, I'm not going to be very happy about it and I'm going to call
your configuration some well-deserved bad names. I don't know much
about spamlist.org, but I'm not impressed so far. My IP isn't listed on
a single spam blacklist listed at <http://www.moensted.dk/spam/> or
<http://www.declude.com/Junkmail/support/ip4r.htm>. (I have seen it
listed on one PacBell blacklist that lists every PacBell IP.)
Fortunately, the SpamAssassin developers have been testing DNS
blacklists for years and include the very best ones in SA, scoring them
with a score optimizer to reduce false positives.
Daniel
--
Daniel Quinlan anti-spam (SpamAssassin), Linux,
http://www.pathname.com/~quinlan/ and open source consulting
Re: How to optimize SA?
Posted by Steve Yuroff <sy...@hiebing.com>.
On Mar 10, 2004, at 5:27 PM, Rick DeNatale wrote:
> On Wed, 2004-03-10 at 08:12, Steve Yuroff wrote:
>
>> That's the first report I've had of the spamlist.org database bouncing
>> mail from a legit source.
>>
>> Are you saying that spamlist.org and the add-in rulesets I'm using are
>> crap? They seemed to be useful, legit tools to me, but I'm here to
>> learn. If it's to be avoided, I'm all for it.
>>
>> My understanding was that the spamlist.org list would cut my
>> processing
>> load at the MTA level... isn't it easier to not accept the mail, than
>> to accept it and then process it for ham/spam?
>
> Unfortunately it seems that a log of ISPs are buying that line and just
> using blacklists to reject e-mail outright. I hate spam, which is why
> I've installed SA locally, and I'm sure that I'm doing a much better
> job
> of filtering spam than my isp is.
>
> Most of the blacklists claim that they have a very low false-positive
> rate, but then most if not all warn against discarding mail based on
> the
> BL alone. I first became aware of the isp/rbl problem when I tried to
> trace down why yahoo groups was periodically suspending sending me mail
> because I was bouncing it. The bounce history showed that my ISP was
> giving hard bounces because the particular yahoo server was blacklisted
> temporarily on spamcop.net. It was no use trying to convince my ISP
> that
> they should not use an RBL to block mail, they LOVED the fact that the
> load on their servers was reduced. The problem is that they had no way
> of knowing whether or not the mail they were rejecting was in fact
> legitimate.
>
> More recently I've had more problems just trying to sign up for some
> support lists on sourceforge, because the registration confirmation
> email never got to me. I finally resorted to opening up my own MTA and
> got the email immediately via this back channel.
>
> Using RBLs as one input to a filtering tool like SA might be a good
> idea, but from what I'm seeing the growing trend towards using them to
> block e-mails is a very bad one. I'm beginning to think that the USPS
> is
> doing a better job of actually delivering my mail than a lot of ISPs
> do.
>
> Sorry for the rant!
>
I've taken the spamlist.org blacklist off of my server since the (I've
been persuaded that it's just too harsh), and I've also removed the
stearns.org blacklist, and Pyzor checks, and my scan times are down to
2-5 seconds, (usually), while my load averages are sitting under 1.
What's odd is that once in a while I see a complete outlier- an 8K
email took 71 seconds to scan, and another that took 47 seconds. The
neighboring entries took under 8 seconds apiece.
How can I determine why some messages take 10x longer to process than
others? These messages seem to be only 1% of the traffic, but I just
want to know what's so odd about them.
Thanks,
Steve.
Re: How to optimize SA?
Posted by Rick DeNatale <ri...@denhaven2.homeip.net>.
On Wed, 2004-03-10 at 08:12, Steve Yuroff wrote:
> That's the first report I've had of the spamlist.org database bouncing
> mail from a legit source.
>
> Are you saying that spamlist.org and the add-in rulesets I'm using are
> crap? They seemed to be useful, legit tools to me, but I'm here to
> learn. If it's to be avoided, I'm all for it.
>
> My understanding was that the spamlist.org list would cut my processing
> load at the MTA level... isn't it easier to not accept the mail, than
> to accept it and then process it for ham/spam?
Unfortunately it seems that a log of ISPs are buying that line and just
using blacklists to reject e-mail outright. I hate spam, which is why
I've installed SA locally, and I'm sure that I'm doing a much better job
of filtering spam than my isp is.
Most of the blacklists claim that they have a very low false-positive
rate, but then most if not all warn against discarding mail based on the
BL alone. I first became aware of the isp/rbl problem when I tried to
trace down why yahoo groups was periodically suspending sending me mail
because I was bouncing it. The bounce history showed that my ISP was
giving hard bounces because the particular yahoo server was blacklisted
temporarily on spamcop.net. It was no use trying to convince my ISP that
they should not use an RBL to block mail, they LOVED the fact that the
load on their servers was reduced. The problem is that they had no way
of knowing whether or not the mail they were rejecting was in fact
legitimate.
More recently I've had more problems just trying to sign up for some
support lists on sourceforge, because the registration confirmation
email never got to me. I finally resorted to opening up my own MTA and
got the email immediately via this back channel.
Using RBLs as one input to a filtering tool like SA might be a good
idea, but from what I'm seeing the growing trend towards using them to
block e-mails is a very bad one. I'm beginning to think that the USPS is
doing a better job of actually delivering my mail than a lot of ISPs do.
Sorry for the rant!
Re: How to optimize SA?
Posted by Steve Yuroff <sy...@hiebing.com>.
On Mar 9, 2004, at 7:55 PM, Daniel Quinlan wrote:
> Okay, hiebing.com rejected my mail. Given how much *crap* has been
> added to their SpamAssassin and MTA configuration, it's no wonder it
> ain't so fast.
>
> ------- start of cut text --------------
> From: Mail Delivery System <Ma...@pathname.com>
> To: quinlan@pathname.com
> Subject: Mail delivery failed: returning message to sender
> Date: Tue, 09 Mar 2004 17:46:49 -0800
>
> This message was created automatically by mail delivery software
> (Exim).
>
> A message that you sent could not be delivered to one or more of its
> recipients. This is a permanent error. The following address(es)
> failed:
>
> syuroff@hiebing.com
> SMTP error from remote mailer after MAIL
> FROM:<qu...@pathname.com>:
> host mail.hiebing.com [64.73.14.137]: 550 5.0.0 UCE Refused by
> www.spamlist.org
> ------- end ----------------------------
>
> Daniel
>
That's the first report I've had of the spamlist.org database bouncing
mail from a legit source.
Are you saying that spamlist.org and the add-in rulesets I'm using are
crap? They seemed to be useful, legit tools to me, but I'm here to
learn. If it's to be avoided, I'm all for it.
My understanding was that the spamlist.org list would cut my processing
load at the MTA level... isn't it easier to not accept the mail, than
to accept it and then process it for ham/spam?
Steve.
Re: How to optimize SA?
Posted by Daniel Quinlan <qu...@pathname.com>.
Okay, hiebing.com rejected my mail. Given how much *crap* has been
added to their SpamAssassin and MTA configuration, it's no wonder it
ain't so fast.
------- start of cut text --------------
From: Mail Delivery System <Ma...@pathname.com>
To: quinlan@pathname.com
Subject: Mail delivery failed: returning message to sender
Date: Tue, 09 Mar 2004 17:46:49 -0800
This message was created automatically by mail delivery software (Exim).
A message that you sent could not be delivered to one or more of its
recipients. This is a permanent error. The following address(es) failed:
syuroff@hiebing.com
SMTP error from remote mailer after MAIL FROM:<qu...@pathname.com>:
host mail.hiebing.com [64.73.14.137]: 550 5.0.0 UCE Refused by www.spamlist.org
------- end ----------------------------
Daniel
--
Daniel Quinlan anti-spam (SpamAssassin), Linux,
http://www.pathname.com/~quinlan/ and open source consulting
Re: Re[4]: [N00b alert]SA in a cpanel environment
Posted by Jim Conner <jc...@lrn.com>.
On Wed, 2004-03-17 at 06:52, lists@arielbusiness.com wrote:
> > A better way to zero out a file is by using '>' notation which will not
> > destroy any file descriptors attached to the file aside from being
> > extremely easy and short.
> >
> > for f in $(find . -name "whatever")
> > do
> > > $f
> > more processing on $f
> > done
>
> Hey Jim thanks for that (though presumably the processing should be done
> on $f BEFORE zeroing it out? ;)
>
:) I love *nix too. Yeah, I apologize for the obfu example. My
example was just showing a fast way to clear out files in a loop.
However, specific to your situation, you will want to massage my idea to
work best with your implementation.
- Jim
> I just love this *nix world - all these obscure abbreviations for things!
>
> cheers,
>
> Tony
--
Jim Conner <jc...@lrn.com>
Re: Re[4]: [N00b alert]SA in a cpanel environment
Posted by li...@arielbusiness.com.
> A better way to zero out a file is by using '>' notation which will not
> destroy any file descriptors attached to the file aside from being
> extremely easy and short.
>
> for f in $(find . -name "whatever")
> do
> > $f
> more processing on $f
> done
Hey Jim thanks for that (though presumably the processing should be done
on $f BEFORE zeroing it out? ;)
I just love this *nix world - all these obscure abbreviations for things!
cheers,
Tony
Re: Re[4]: [N00b alert]SA in a cpanel environment
Posted by Jim Conner <jc...@lrn.com>.
On Tue, 2004-03-16 at 23:14, lists@arielbusiness.com wrote:
> > TB> Hey cool, done that now. Just looked at the headers of a message
> > TB> received which says "autolearn=ham" This was a message from the SA
> > TB> group funnily enough - presumably that is correct?
> >
> > Unless that message included spam samples, then no problem.
> >
> > I suggest you set your non-spam auto-learn threshold to -0.01 to make
> > sure that spam that hits no rules is not accidentally learned as ham.
>
> errr...what? who? where? how???
> Also, loooking at more headers - if they say "autolearn=no", does that
> just mean SA had no idea if it was spam or ham, or does it just mean that
> autolearn is off and I Was looking at an old message? ;)else?
>
> > My understanding is that each domain with a $HOME will have one
> > $HOME/.spamassassin directory, and the bayes database built there will
> > apply to all users@domain for that domain.
>
> Cool. That does indeed seem to be the case - my mailboxes were
> refreshingly free of spam this morning - hurrah!
>
> > cp /dev/null $file
> > or
> > cat </dev/null >$file
> > are two methods I've used to empty files.
>
> okay will do that - is there any advantage of one over the other apart
> from less typing? ;)
>
A better way to zero out a file is by using '>' notation which will not
destroy any file descriptors attached to the file aside from being
extremely easy and short.
for f in $(find . -name "whatever")
do
> $f
more processing on $f
done
> > TB> - is my first ever shell script!:
> [..]
> > TB> Any obvious flaws there guys, or something I could do better? It
> >
> > Looks good to me. I wouldn't cat them all into one file first, since my
> > understanding is that the shorter/quicker sa-learn runs are better (less
> > chance they'll block bayes update by incoming email and auto-learn).
>
> okay, thanks m8. You cat them in your script though don't you?
>
> > TB> If the former, then presumably my script would be better off
> > contatenating
> > TB> the spam and ham files before passing them to a single run of
> > sa-learn?
> >
> > I run my scripts once an hour.
>
> Blimey - do you get THAT much spam? ;)
>
> > You need 200+ spams and 200+ hams before Bayes takes effect and starts
> > applying its scores to your emails. It then remains effective unless you
> > drop below those numbers (such as by deleting the database files and
> > starting over). That has nothing to do with sa-learn. The more often
> > sa-learn runs, the more current your bayes database is.
>
> Okay, thanks.
> For ham, do you just copy everything from your inbox (apart from spam not
> caught) or is there stuff you WOULDN'T put through the spam filter? eg,
> all the posts to this list?
> I am on a number of lists and the volume would make perfect ham material,
> but I'd be worried sometimes that the content wouldn't and certain
> characteristics - eg being sent to a large no. of people, me not being
> explicitly set as a recipient.
> For spam, is there any value in passing already identified spam (sent to
> the spambox thru sa-learn?
>
> thanks again,
>
> Tony
--
Jim Conner <jc...@lrn.com>
Re: Re[4]: [N00b alert]SA in a cpanel environment
Posted by li...@arielbusiness.com.
> TB> Hey cool, done that now. Just looked at the headers of a message
> TB> received which says "autolearn=ham" This was a message from the SA
> TB> group funnily enough - presumably that is correct?
>
> Unless that message included spam samples, then no problem.
>
> I suggest you set your non-spam auto-learn threshold to -0.01 to make
> sure that spam that hits no rules is not accidentally learned as ham.
errr...what? who? where? how???
Also, loooking at more headers - if they say "autolearn=no", does that
just mean SA had no idea if it was spam or ham, or does it just mean that
autolearn is off and I Was looking at an old message? ;)else?
> My understanding is that each domain with a $HOME will have one
> $HOME/.spamassassin directory, and the bayes database built there will
> apply to all users@domain for that domain.
Cool. That does indeed seem to be the case - my mailboxes were
refreshingly free of spam this morning - hurrah!
> cp /dev/null $file
> or
> cat </dev/null >$file
> are two methods I've used to empty files.
okay will do that - is there any advantage of one over the other apart
from less typing? ;)
> TB> - is my first ever shell script!:
[..]
> TB> Any obvious flaws there guys, or something I could do better? It
>
> Looks good to me. I wouldn't cat them all into one file first, since my
> understanding is that the shorter/quicker sa-learn runs are better (less
> chance they'll block bayes update by incoming email and auto-learn).
okay, thanks m8. You cat them in your script though don't you?
> TB> If the former, then presumably my script would be better off
> contatenating
> TB> the spam and ham files before passing them to a single run of
> sa-learn?
>
> I run my scripts once an hour.
Blimey - do you get THAT much spam? ;)
> You need 200+ spams and 200+ hams before Bayes takes effect and starts
> applying its scores to your emails. It then remains effective unless you
> drop below those numbers (such as by deleting the database files and
> starting over). That has nothing to do with sa-learn. The more often
> sa-learn runs, the more current your bayes database is.
Okay, thanks.
For ham, do you just copy everything from your inbox (apart from spam not
caught) or is there stuff you WOULDN'T put through the spam filter? eg,
all the posts to this list?
I am on a number of lists and the volume would make perfect ham material,
but I'd be worried sometimes that the content wouldn't and certain
characteristics - eg being sent to a large no. of people, me not being
explicitly set as a recipient.
For spam, is there any value in passing already identified spam (sent to
the spambox thru sa-learn?
thanks again,
Tony
Re: Re[8]: [N00b alert]SA in a cpanel environment
Posted by li...@arielbusiness.com.
Hi Bob,
> lac> sorry, do you mean my local (to the account) config, or the config
> for SA
> lac> on the server generally?
>
> That's up to you. system-wide or user-specific depends on how you want to
> use it.
Okey doke, tnx. I will now go look up the docs to find out exactly what
it means so I can decide where I want to put it ;)
> lac> Also, I see you using yet another method of emptying the mailbox:
> lac> cat ~/mynull > sa.learn.spam
>
> lac> Any particular reason for using this one?
>
> Some email programs want specific header emails at the top of their
> folders. This is one way of providing one such initial header email.
Ahhhh.....fair nuff.
tnx again very much for your help.
Tony
Re[8]: [N00b alert]SA in a cpanel environment
Posted by Robert Menschel <Ro...@Menschel.net>.
Hello Tony,
Wednesday, March 17, 2004, 4:57:09 AM, you wrote:
>> I use:
>>> auto_learn_threshold_nonspam -2
>>> bayes_auto_learn_threshold_nonspam -2
>> I forget which version applies to 2.5x and which to 2.6x -- adapt to the
>> score you want to use as a threshold, and put it into your local config
>> file (eg: local.cf or whatever).
lac> sorry, do you mean my local (to the account) config, or the config for SA
lac> on the server generally?
That's up to you. system-wide or user-specific depends on how you want to
use it.
lac> Sorry, yes. What's this corpus file for?
a) to provide resources with which to re-learn spam and ham after losing
a Bayes database.
b) to provide emails against which we can test new custom rules we
develop (per http://www.exit0.us/ )
c) to provide emails against which new rules created by the devs can be
run, to assist with the setting of scores for those rules.
lac> Also, I see you using yet another method of emptying the mailbox:
lac> cat ~/mynull > sa.learn.spam
lac> Any particular reason for using this one?
Some email programs want specific header emails at the top of their
folders. This is one way of providing one such initial header email.
Bob Menschel
Re: Re[6]: [N00b alert]SA in a cpanel environment
Posted by li...@arielbusiness.com.
>>> I suggest you set your non-spam auto-learn threshold to -0.01 to make
>>> sure that spam that hits no rules is not accidentally learned as ham.
>
> tac> errr...what? who? where? how???
>
> I use:
>> auto_learn_threshold_nonspam -2
>> bayes_auto_learn_threshold_nonspam -2
> I forget which version applies to 2.5x and which to 2.6x -- adapt to the
> score you want to use as a threshold, and put it into your local config
> file (eg: local.cf or whatever).
sorry, do you mean my local (to the account) config, or the config for SA
on the server generally?
> tac> Also, loooking at more headers - if they say "autolearn=no", does
[..]
> No, autolearn=no simply means the email didn't score high enough (as
> spam) or low enough (as non-spam) to be auto-learned. It means auto-learn
> is on, but the email message didn't qualify.
gotchya, tnx!>
>>> cp /dev/null $file
>>> or
>>> cat </dev/null >$file
>>> are two methods I've used to empty files.
>
> tac> okay will do that - is there any advantage of one over the other
> apart
> tac> from less typing? ;)
>
> None that we can measure.
LOL, fair nufff, just wondering why you use BOTH.
>>> Looks good to me. I wouldn't cat them all into one file first, since
> tac> okay, thanks m8. You cat them in your script though don't you?
>
> No. My commands are:
>> sa-learn --spam --mbox sa.learn.spam # do the sa-learn
>> ls -lF `pwd`/sa.learn.spam # record this file in my log
>> cat sa.learn.spam >>~/mail/cw-spam/inbox # append to my corpus
>> cat ~/mynull > sa.learn.spam # empty the mailbox
Sorry, yes. What's this corpus file for?
Also, I see you using yet another method of emptying the mailbox:
cat ~/mynull > sa.learn.spam
Any particular reason for using this one?
>>> I run my scripts once an hour.
>
> tac> Blimey - do you get THAT much spam? ;)
>
> 7-8k spam a week, and will probably hit 9k around June.
No wonder you run it so often ;)
> I do not sa-learn the SA mailing lists, nor any other mail which contains
> samples of spam, nor discussions of spam. Otherwise I sa-learn
> everything.
Thanks for that. Funnily enough, looking in my spambox, sa had been
tagging certain posts from this list as SPAM because they containted
samples of spam.....must add the list to my whitelist ;)
> b) For simplicity of management, I dump spam from all three domains into
> one spamtrap. It's easier for me then to sa-learn all of them, rather
> than keep the already learned spam separate.
good idea! You've been a great help Bob, many thanks.
Tony
Re[6]: [N00b alert]SA in a cpanel environment
Posted by Robert Menschel <Ro...@Menschel.net>.
Hello tony,
Tuesday, March 16, 2004, 11:08:43 PM, you wrote:
>> TB> Hey cool, done that now. Just looked at the headers of a message
>> TB> received which says "autolearn=ham" This was a message from the SA
>> TB> group funnily enough - presumably that is correct?
>>
>> Unless that message included spam samples, then no problem.
>>
>> I suggest you set your non-spam auto-learn threshold to -0.01 to make
>> sure that spam that hits no rules is not accidentally learned as ham.
tac> errr...what? who? where? how???
I use:
> auto_learn_threshold_nonspam -2
> bayes_auto_learn_threshold_nonspam -2
I forget which version applies to 2.5x and which to 2.6x -- adapt to the
score you want to use as a threshold, and put it into your local config
file (eg: local.cf or whatever).
tac> Also, loooking at more headers - if they say "autolearn=no", does that
tac> just mean SA had no idea if it was spam or ham, or does it just mean that
tac> autolearn is off and I Was looking at an old message? ;)else?
No, autolearn=no simply means the email didn't score high enough (as
spam) or low enough (as non-spam) to be auto-learned. It means auto-learn
is on, but the email message didn't qualify.
>> My understanding is that each domain with a $HOME will have one
>> $HOME/.spamassassin directory, and the bayes database built there will
>> apply to all users@domain for that domain.
tac> Cool. That does indeed seem to be the case - my mailboxes were
tac> refreshingly free of spam this morning - hurrah!
>> cp /dev/null $file
>> or
>> cat </dev/null >$file
>> are two methods I've used to empty files.
tac> okay will do that - is there any advantage of one over the other apart
tac> from less typing? ;)
None that we can measure.
>> TB> - is my first ever shell script!:
tac> [..]
>> TB> Any obvious flaws there guys, or something I could do better? It
>>
>> Looks good to me. I wouldn't cat them all into one file first, since my
>> understanding is that the shorter/quicker sa-learn runs are better (less
>> chance they'll block bayes update by incoming email and auto-learn).
tac> okay, thanks m8. You cat them in your script though don't you?
No. My commands are:
> sa-learn --spam --mbox sa.learn.spam # do the sa-learn
> ls -lF `pwd`/sa.learn.spam # record this file in my log
> cat sa.learn.spam >>~/mail/cw-spam/inbox # append to my corpus
> cat ~/mynull > sa.learn.spam # empty the mailbox
>> TB> If the former, then presumably my script would be better off
>> contatenating
>> TB> the spam and ham files before passing them to a single run of
>> sa-learn?
>>
>> I run my scripts once an hour.
tac> Blimey - do you get THAT much spam? ;)
7-8k spam a week, and will probably hit 9k around June.
>> You need 200+ spams and 200+ hams before Bayes takes effect and starts
>> applying its scores to your emails. It then remains effective unless you
>> drop below those numbers (such as by deleting the database files and
>> starting over). That has nothing to do with sa-learn. The more often
>> sa-learn runs, the more current your bayes database is.
tac> Okay, thanks.
tac> For ham, do you just copy everything from your inbox (apart from spam not
tac> caught) or is there stuff you WOULDN'T put through the spam filter? eg,
tac> all the posts to this list?
I do not sa-learn the SA mailing lists, nor any other mail which contains
samples of spam, nor discussions of spam. Otherwise I sa-learn
everything.
tac> I am on a number of lists and the volume would make perfect ham material,
tac> but I'd be worried sometimes that the content wouldn't and certain
tac> characteristics - eg being sent to a large no. of people, me not being
tac> explicitly set as a recipient.
If you (or your people) ever get non-spam from the same people who use
those lists, discussing the same topics, then learning them will be a big
help (avoid FPs).
tac> For spam, is there any value in passing already identified spam (sent to
tac> the spambox thru sa-learn?
a) If you do multiple domains as I do, then it's easier to simply feed
all spam into all three domains rather than figure out where it has
already been learned.
b) For simplicity of management, I dump spam from all three domains into
one spamtrap. It's easier for me then to sa-learn all of them, rather
than keep the already learned spam separate.
Bob Menschel
Re[4]: [N00b alert]SA in a cpanel environment
Posted by Robert Menschel <Ro...@Menschel.net>.
Hello Tony,
Tuesday, March 16, 2004, 2:45:59 PM, you wrote:
>> If you have access to the /etc/mail/spamassassin/local.cf
>> file (may be in a different directory according to how SA is
>> called), then you can add the parameter
>> > bayes_auto_learn 1
TB> Hey cool, done that now. Just looked at the headers of a message
TB> received which says "autolearn=ham" This was a message from the SA
TB> group funnily enough - presumably that is correct?
Unless that message included spam samples, then no problem.
I suggest you set your non-spam auto-learn threshold to -0.01 to make
sure that spam that hits no rules is not accidentally learned as ham.
TB> I managed to get sa-learn to work for a non-root account by deleting the
TB> bayes* files as you suggested.
TB> Presumably the bayes database applies then to any user@domain for the
TB> userid I run it under?
My understanding is that each domain with a $HOME will have one
$HOME/.spamassassin directory, and the bayes database built there will
apply to all users@domain for that domain.
TB> I've had a look at your script and it's given me some ideas thanks - I have
TB> written a script which will look for all files called learn_spam or
TB> learn_ham and run sa-learn on them, then "empties" the files by removing
TB> them and touching them (is there a better way?)
cp /dev/null $file
or
cat </dev/null >$file
are two methods I've used to empty files.
TB> I know nothing about shell programming other than what I have picked up from
TB> Bob's script and google, so forgive if it's a little rough around the edges
TB> - is my first ever shell script!:
TB> ====================================
TB> #!/bin/sh
TB> if [ $1 -eq "d" ] ; then
TB> SARGS="--showdots"
TB> fi
TB> echo "Learning SPAM"
TB> for FILE in `find $HOME -name learn_spam -print`
TB> do
TB> echo "Processing $FILE"
TB> sa-learn --spam --mbox $FILE $SARGS
TB> rm $FILE
TB> touch $FILE
TB> done
TB> echo "Learning HAM"
TB> for FILE in `find $HOME -name learn_ham -print`
TB> do
TB> echo "Processing $FILE"
TB> sa-learn --ham --mbox $FILE $SARGS
TB> rm $FILE
TB> touch $FILE
TB> done
TB> echo "Done"
TB> ====================================
TB> Any obvious flaws there guys, or something I could do better? It *seems*
TB> to work okay anyway.
TB> Should I bung them all into one file first????
Looks good to me. I wouldn't cat them all into one file first, since my
understanding is that the shorter/quicker sa-learn runs are better (less
chance they'll block bayes update by incoming email and auto-learn).
TB> The other thing is, how often should I run it - I've seen it mentioned
TB> before that you need about 200 spams and 200 hams for sa-learn to be
TB> effective - does this mean 200 _per run_ or that you need to have learned
TB> about that number in total for it to be effective?
TB> If the former, then presumably my script would be better off contatenating
TB> the spam and ham files before passing them to a single run of sa-learn?
I run my scripts once an hour.
You need 200+ spams and 200+ hams before Bayes takes effect and starts
applying its scores to your emails. It then remains effective unless you
drop below those numbers (such as by deleting the database files and
starting over). That has nothing to do with sa-learn. The more often
sa-learn runs, the more current your bayes database is.
Bob Menschel
RE: Re[2]: SA in a cpanel environment - NOW tie failed: File exists
Posted by Tony Butler <li...@arielbusiness.com>.
> If you have access to the /etc/mail/spamassassin/local.cf file (may be
> in a different directory according to how SA is called), then you can
> add the parameter
> > bayes_auto_learn 1
Hey cool, done that now. Just looked at the headers of a message received
which says "autolearn=ham"
This was a message from the SA group funnily enough - presumably that is
correct?
I managed to get sa-learn to work for a non-root account by deleting the
bayes* files as you suggested.
Presumably the bayes database applies then to any user@domain for the userid
I run it under?
I've had a look at your script and it's given me some ideas thanks - I have
written a script which will look for all files called learn_spam or
learn_ham and run sa-learn on them, then "empties" the files by removing
them and touching them (is there a better way?)
I know nothing about shell programming other than what I have picked up from
Bob's script and google, so forgive if it's a little rough around the edges
- is my first ever shell script!:
====================================
#!/bin/sh
if [ $1 -eq "d" ] ; then
SARGS="--showdots"
fi
echo "Learning SPAM"
for FILE in `find $HOME -name learn_spam -print` do
echo "Processing $FILE"
sa-learn --spam --mbox $FILE $SARGS
rm $FILE
touch $FILE
done
echo "Learning HAM"
for FILE in `find $HOME -name learn_ham -print` do
echo "Processing $FILE"
sa-learn --ham --mbox $FILE $SARGS
rm $FILE
touch $FILE
done
echo "Done"
====================================
Any obvious flaws there guys, or something I could do better? It *seems*
to work okay anyway.
Should I bung them all into one file first????
The other thing is, how often should I run it - I've seen it mentioned
before that you need about 200 spams and 200 hams for sa-learn to be
effective - does this mean 200 _per run_ or that you need to have learned
about that number in total for it to be effective?
If the former, then presumably my script would be better off contatenating
the spam and ham files before passing them to a single run of sa-learn?
> I like salt-free sunflower seeds...
They're on their way!
Cheers,
Tony
Re[2]: SA in a cpanel environment - NOW tie failed: File exists
Posted by Robert Menschel <Ro...@Menschel.net>.
Hello Tony,
Monday, March 15, 2004, 3:02:23 PM, you wrote:
>> If you have access to the /etc/mail/spamassassin/local.cf
>> file (may be in a different directory according to how SA is
>> called), then you can add the parameter
>> > bayes_auto_learn 1
>> to that file.
TB> Okey doke, done that...now how can I tell if it worked???
TB> Do I need to restart something maybe?
If you're running spamd/spamc or equivalent, you normally need to restart
in order to have parameter changes take effect.
If auto-learn is turned on, then the bayes databases will be updated by
emails which exceed the auto-learn thresholds in either direction. I find
three bayes database files in my $home/.spamassassin directory:
bayes_journal, bayes_seen, and bayes_toks. All three are updated during
the auto-learn process.
>> T> How much do you want for a copy of your scripts? ;)
>>
>> I like salt-free sunflower seeds...
TB> LOL, send me your address and they're on their way ;)
Bob Menschel
P. O. Box 2369
Citrus Heights, CA 95611
Donations gratefully accepted from all sources. Not tax deductible.
TB> Had a look at the script and can even reasonably understand it - is not as
TB> complex as I expected - hurrah!
TB> That's the good news.
TB> The bad news is, when I try (not using your script _just_ yet) to run
TB> sa-learn NOT logged on as root (I know root works) I get an error, which
TB> tells me to run it using -D, which produces this:
TB> -bash-2.05b$ sa-learn --spam --showdots --mbox
TB> mail/arielbusiness.com/abutler/spam -D
TB> debug: Score set 0 chosen.
TB> ...
TB> debug: using "/home/arielbus/.spamassassin/user_prefs" for user prefs file
TB> debug: bayes: 30704 tie-ing to DB file R/O
TB> /home/arielbus/.spamassassin/bayes_toks
TB> Cannot open bayes databases /home/arielbus/.spamassassin/bayes_* R/O: tie
TB> failed:
TB> ...
TB> Cannot open bayes databases /home/arielbus/.spamassassin/bayes_* R/W: tie
TB> failed: File exists
TB> ...
TB> Cannot open bayes databases /home/arielbus/.spamassassin/bayes_* R/W: tie
TB> failed: File exists
TB> ...
TB> Question: what does it all mean & how can I fix it???????
Looks to me like you either have a permissions problem with those bayes_*
files, or the files and software are incompatible.
When you issue the command above, what userid are you, who owns those
files, and what are their permissions? If owned by root and changable
only by root, that could cause this problem.
Is this an upgrade from an earlier (2.5x or older) version? If so, did
you follow the install instructions for updating the database?
Do you have the correct perl DB package installed at a reasonably current
level?
When my host moved my domains from 2.5x to 2.6x, two of my Bayes
databases migrated fine, and one ended up "broken" with symptoms not
dissimilar from what you show. I deleted those bayes_* files, waited for
SA to recreate them during normal processing, and then used sa-learn to
reload them with current emails. Worked fine.
Bob Menschel
Re[2]: [N00b alert]SA in a cpanel environment
Posted by Robert Menschel <Ro...@Menschel.net>.
Hello Tony,
Thursday, March 11, 2004, 11:25:59 PM, you wrote:
>>> I would like to do the following:
>>
>>> 1. Turn auto learn on (unless this is not recommended???).
>>
>> In my cpanel installations auto-learn is automatic.
T> Any idea how I can turn this on? It is turned off on my server, and those
T> of others using the same dedicated server provider - and they don't do it
T> themselves, so I assume it is OFF by default in recent versions of
T> CPanel....
If you have access to the /etc/mail/spamassassin/local.cf file (may be in
a different directory according to how SA is called), then you can add
the parameter
> bayes_auto_learn 1
to that file. See http://www.spamassassin.org/doc/Mail_SpamAssassin_Conf.html
>>> 2. Allow my clients to run sa-learn if they wish (they do not have ssh
>>> access, but can run cron jobs), ideallty on a regular automated basis
>>
>> I use cron scripts which look for files named newspam.txt and
>> newham.txt in my domain root directories, and if found, run sa-learn
>> against and then delete the files. I create these files by FTP upload
>> from my email client. Works great.
T> Cool. Do you run these for the whole server or just for YOUR domain?
Just my domain (it's the only area I have permissions for). It would not
be hard to use similar logic to scan the entire server.
>> Only requirement is an email client that will output
>> intelligent/useful files (I use mbox format).
T> ditto.
>> I have users that use horde for webmail. I've created hamlearn and
>> spamlearn folders for them, and my cron scripts look to see if those
>> folders are non-empty. If so, it runs sa-learn against those folders
>> and then empties them.
T> Sounds good, just the sort of thing I'm after.
>>> 3. Run sa-learn on a regular, automated basis for the entire server
>>
>> That can be a challenge.
T> running it for the entire server, or running it automatically?
Running it for the entire server, and allowing people to have their own
customized database.
>> sa-learn runs best for individual users and/or individual systems.
>> While a system-wide bayes is better than none, you won't want to
>> interfere with those users/domains that are able to do their own bayes
>> work.
T> Can you tell me how to run it for the entire server?
T> Why is it better to have individual ones for each user - becuause they
T> have different ideas of what constitutes spam?
Not only different ideas of what constitutes spam, but also actually
different spam, and more importantly, different ham. The domain used by a
rape support group will have VERY different ham than the domain used by a
mortgage banker.
The way to run it for the entire server would be to run it as a specific
userid, and to set up local.cf to ensure that EVERYONE uses that specific
userid's bayes database. Others can tell you how to do that.
>> As an end-user, I can't help you much with the cpanel setup, but I can
>> help you with the types of things I have managed to do here.
T> Sounds like you're a lot more than an end-user Bob.
T> You are doing a heck of a lot that I would like to be able to do.
T> Apart from the lack of SA/cpanel knowledge, I am a *indows man not a *nix
T> one, so cron jobs/perl scripts/shell scripts are all pretty new to me.
T> How much do you want for a copy of your scripts? ;)
I like salt-free sunflower seeds...
Bob Menschel
Re: [N00b alert]SA in a cpanel environment
Posted by li...@arielbusiness.com.
Hi Bob,
> Hello lists,
You can call me Tony ;)
Many thanks for responding to my plea.
>> a "normal" install of an app does not apply to the cpanelled version :/
>
> Understood. I'm a domain master on three similar machines (three
> different domains), all using SA under cpanel's installation. Works
> well for me (though I've strengthened SA for myself more than most
> others would be able to).
Ah, a guru then ;)
>> I would like to do the following:
>
>> 1. Turn auto learn on (unless this is not recommended???).
>
> In my cpanel installations auto-learn is automatic.
Any idea how I can turn this on? It is turned off on my server, and those
of others using the same dedicated server provider - and they don't do it
themselves, so I assume it is OFF by default in recent versions of
CPanel....
>> 2. Allow my clients to run sa-learn if they wish (they do not have ssh
>> access, but can run cron jobs), ideallty on a regular automated basis
>
> I use cron scripts which look for files named newspam.txt and
> newham.txt in my domain root directories, and if found, run sa-learn
> against and then delete the files. I create these files by FTP upload
> from my email client. Works great.
Cool. Do you run these for the whole server or just for YOUR domain?
> Only requirement is an email client that will output
> intelligent/useful files (I use mbox format).
ditto.
> I have users that use horde for webmail. I've created hamlearn and
> spamlearn folders for them, and my cron scripts look to see if those
> folders are non-empty. If so, it runs sa-learn against those folders
> and then empties them.
Sounds good, just the sort of thing I'm after.
>> 3. Run sa-learn on a regular, automated basis for the entire server
>
> That can be a challenge.
running it for the entire server, or running it automatically?
>> and certainly I haven't noticed a drop-off in spam, which is why I think
>> it was for the root account that this was run rather than for the whole
>> server.
>
> I agree -- sa-learn normally runs for whatever user runs it.
> sa-learn runs best for individual users and/or individual systems.
> While a system-wide bayes is better than none, you won't want to
> interfere with those users/domains that are able to do their own bayes
> work.
Can you tell me how to run it for the entire server?
Why is it better to have individual ones for each user - becuause they
have different ideas of what constitutes spam?
> As an end-user, I can't help you much with the cpanel setup, but I can
> help you with the types of things I have managed to do here.
Sounds like you're a lot more than an end-user Bob.
You are doing a heck of a lot that I would like to be able to do.
Apart from the lack of SA/cpanel knowledge, I am a *indows man not a *nix
one, so cron jobs/perl scripts/shell scripts are all pretty new to me.
How much do you want for a copy of your scripts? ;)
many thanks,
Tony
Re: [N00b alert]SA in a cpanel environment
Posted by Bob Menschel <Ro...@Menschel.net>.
Hello lists,
Wednesday, March 10, 2004, 11:12:32 PM, you wrote:
> Hi,
> Excuse the n00b questions, but I cannot find the answers to these
> questions anywhere and have tried various cpanel forums.
> I run small hosting business on a red hat box with cpanel/whm and cpanel
> tend to muck around with things a bit so sometimes advice that appplies to
> a "normal" install of an app does not apply to the cpanelled version :/
Understood. I'm a domain master on three similar machines (three
different domains), all using SA under cpanel's installation. Works
well for me (though I've strengthened SA for myself more than most
others would be able to).
> I would like to do the following:
> 1. Turn auto learn on (unless this is not recommended???).
In my cpanel installations auto-learn is automatic.
> 2. Allow my clients to run sa-learn if they wish (they do not have ssh
> access, but can run cron jobs), ideallty on a regular automated basis
I use cron scripts which look for files named newspam.txt and
newham.txt in my domain root directories, and if found, run sa-learn
against and then delete the files. I create these files by FTP upload
from my email client. Works great.
Only requirement is an email client that will output
intelligent/useful files (I use mbox format).
I have users that use horde for webmail. I've created hamlearn and
spamlearn folders for them, and my cron scripts look to see if those
folders are non-empty. If so, it runs sa-learn against those folders
and then empties them.
> 3. Run sa-learn on a regular, automated basis for the entire server
That can be a challenge.
> Regarding 3. above, I have logged on as root and successfully run sa-learn
> against a couple of spam and ham folders......but I think this may have
> been for the root user rather than for the server as a whole.
> I just did something like:
> sa-learn --showdots --spam --mbox <path to mailbox file>
> without specifying any paths to config files or any such thing like that,
> and certainly I haven't noticed a drop-off in spam, which is why I think
> it was for the root account that this was run rather than for the whole
> server.
I agree -- sa-learn normally runs for whatever user runs it.
sa-learn runs best for individual users and/or individual systems.
While a system-wide bayes is better than none, you won't want to
interfere with those users/domains that are able to do their own bayes
work.
As an end-user, I can't help you much with the cpanel setup, but I can
help you with the types of things I have managed to do here.
Bob Menschel
[N00b alert]SA in a cpanel environment
Posted by li...@arielbusiness.com.
Hi,
Excuse the n00b questions, but I cannot find the answers to these
questions anywhere and have tried various cpanel forums.
I run small hosting business on a red hat box with cpanel/whm and cpanel
tend to muck around with things a bit so sometimes advice that appplies to
a "normal" install of an app does not apply to the cpanelled version :/
I would like to do the following:
1. Turn auto learn on (unless this is not recommended???).
2. Allow my clients to run sa-learn if they wish (they do not have ssh
access, but can run cron jobs), ideallty on a regular automated basis
3. Run sa-learn on a regular, automated basis for the entire server
Regarding 3. above, I have logged on as root and successfully run sa-learn
against a couple of spam and ham folders......but I think this may have
been for the root user rather than for the server as a whole.
I just did something like:
sa-learn --showdots --spam --mbox <path to mailbox file>
without specifying any paths to config files or any such thing like that,
and certainly I haven't noticed a drop-off in spam, which is why I think
it was for the root account that this was run rather than for the whole
server.
Also, ideally, once sa-learn is finished, I would like it to empty the
spam and ham folders.
thank you kindly in advance for any help you can give,
Tony
RE: How to optimize SA?
Posted by Bret Miller <br...@wcg.org>.
> > I've been doing some debug runs on the included
> sample-spam.txt, and I
> > see that's where the biggest pause happens right before that line.
> > Maybe a shared bayes database for my whole company isn't
> going to work
> > anymore?
>
> I believe individual bayes databases are probably faster, but
> other people seem to do quite okay with shared bayes databases.
We're running more messages through than that with a shared bayes
database on an AMD 700 with 768MB ram on Windows. Not a fair comparison,
I know, but I'd doubt that bayes database contention is much of an
issue.
Bret
Re: How to optimize SA?
Posted by Daniel Quinlan <qu...@pathname.com>.
Steve Yuroff <sy...@hiebing.com> writes:
> Would lock contention issues look like this?:
>
> debug: bayes: 8340 tie-ing to DB file R/O
> /home/users/sharedspam/.spamassassin/bayes_toks
> debug: bayes: 8340 tie-ing to DB file R/O
> /home/users/sharedspam/.spamassassin/bayes_seen
No, that's just R/O locking.
> I've been doing some debug runs on the included sample-spam.txt, and I
> see that's where the biggest pause happens right before that line.
> Maybe a shared bayes database for my whole company isn't going to work
> anymore?
I believe individual bayes databases are probably faster, but other
people seem to do quite okay with shared bayes databases.
Daniel
--
Daniel Quinlan anti-spam (SpamAssassin), Linux,
http://www.pathname.com/~quinlan/ and open source consulting
Re: How to optimize SA?
Posted by Steve Yuroff <sy...@hiebing.com>.
On Mar 9, 2004, at 7:46 PM, Daniel Quinlan wrote:
> Steve Yuroff <sy...@hiebing.com> writes:
>
>> OK, I must be doing something wrong! This is a Qube 3 Pro, 384M, 40
>> users, shared Bayes.
>
> You could be having some lock contention issues with a read-write
> shared
> Bayes, but I doubt that too.
>
>>
Would lock contention issues look like this?:
debug: bayes: 8340 tie-ing to DB file R/O
/home/users/sharedspam/.spamassassin/bayes_toks
debug: bayes: 8340 tie-ing to DB file R/O
/home/users/sharedspam/.spamassassin/bayes_seen
I've been doing some debug runs on the included sample-spam.txt, and I
see that's where the biggest pause happens right before that line.
Maybe a shared bayes database for my whole company isn't going to work
anymore?
Thanks all for your time helping me out. I greatly appreciate it.
Steve.
Re: How to optimize SA?
Posted by Daniel Quinlan <qu...@pathname.com>.
Steve Yuroff <sy...@hiebing.com> writes:
> OK, I must be doing something wrong! This is a Qube 3 Pro, 384M, 40
> users, shared Bayes.
You could be having some lock contention issues with a read-write shared
Bayes, but I doubt that too.
> SA 2.63. BigEvil, BlackHair, evilnumbers, popcorn all included. I
> see under 2K messages/day, about 70-80% spam.
It might be that one of the rule sets you've added that is making it
slow(er).
> I need to run the test message through spamassassin -D and see where
> it hangs.... but I can't remember where the default message lives.
> Anyone want to jog my memory?
Depends on how it was installed. It's better to use a few random emails
received from the outside world and sent to your site, save them in Unix
mbox mode and then test.
Daniel
--
Daniel Quinlan anti-spam (SpamAssassin), Linux,
http://www.pathname.com/~quinlan/ and open source consulting
Re: How to optimize SA?
Posted by Steve Yuroff <sy...@hiebing.com>.
On Mar 9, 2004, at 3:57 PM, Scott Blomquist wrote:
>
>
> Steve Yuroff wrote:
>
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>> I've been running SA on a Cobalt Qube since 2.42 was current, and
>> it's been a wonderful solution. Recently I moved to running spamd -d
>> to because of the increased load the increase in spam was putting on
>> the server. Now that's not cutting it- they're hitting me so hard
>> I'm seeing processor loads in the 30's, and my poor little Qube is
>> buckling under the load- I can't ssh in, email exchanges time out,
>> ect.
>> What else can I do to make SA run as quickly as possible- besides the
>> obvious "buy a bigger server"?
>> Thanks,
>
> Hi Steve,
> What model Qube, how many users, what version of SA?
> I'm running a Q3 w/512M and 56 users all individual Bayes and
> User_prefs and my load is .1. Sometimes it gets to 1.5 if I'm under a
> spamstorm but usually it isn't even breathing hard. We're averaging
> 3200 messages a day with spam accounting for 66% of the load.
> Scott
>
OK, I must be doing something wrong! This is a Qube 3 Pro, 384M, 40
users, shared Bayes. SA 2.63. BigEvil, BlackHair, evilnumbers,
popcorn all included. I see under 2K messages/day, about 70-80% spam.
I need to run the test message through spamassassin -D and see where it
hangs.... but I can't remember where the default message lives. Anyone
want to jog my memory?
Re: How to optimize SA?
Posted by Scott Blomquist <sc...@innertraditions.com>.
Steve Yuroff wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> I've been running SA on a Cobalt Qube since 2.42 was current, and it's
> been a wonderful solution. Recently I moved to running spamd -d to
> because of the increased load the increase in spam was putting on the
> server. Now that's not cutting it- they're hitting me so hard I'm
> seeing processor loads in the 30's, and my poor little Qube is buckling
> under the load- I can't ssh in, email exchanges time out, ect.
>
> What else can I do to make SA run as quickly as possible- besides the
> obvious "buy a bigger server"?
>
> Thanks,
>
Hi Steve,
What model Qube, how many users, what version of SA?
I'm running a Q3 w/512M and 56 users all individual Bayes and User_prefs
and my load is .1. Sometimes it gets to 1.5 if I'm under a spamstorm but
usually it isn't even breathing hard. We're averaging 3200 messages a
day with spam accounting for 66% of the load.
Scott
--
Scott V. Blomquist,A-SA-CN-NRK TINLC(tm) #2598
ITI/Bear&Co Rochester, VT
802-767-3174(v) 802-767-3726(f)
"Any technology sufficiently advanced is indistinguishable from Magic."
A. C. Clarke
Re: How to optimize SA?
Posted by Daniel Quinlan <qu...@pathname.com>.
Justin Mason wrote:
>> FWIW, though, SpamAssassin 2.6x has a very sophisticated DNSBL lookup
>> algorithm which (a) runs all queries in parallel and (b) aborts ones
>> that are taking significantly longer than the others, reducing problems
>> if one or two hosts go down. So nowadays this should be *more* efficient
>> run from SpamAssassin than from the MTA.
Jason Haar <Ja...@trimble.co.nz> writes:
> I don't doubt it. But that means spamc hangs around for up to 10 seconds
> awaiting a response on a system that runs "spamd -m XX" - i.e. XX max
> processes.
It's more like 3 seconds when some DNSBL are timing out, not 10.
Sometimes, the MX checking tests can be slow, but you can turn those off
independently of the DNSBL (RBL)tests.
> If you ran RBL checks out on the SMTP server, then you may be
> inefficient, but at least that 25K SMTP process is dealing with it
> rather than a 28M spamd process...
You realize that fork() in Linux (and BSD too, I'm sure) doesn't copy
every page? 28M virtual size != the actual additional memory used by a
spamd process.
Nevermind the fact that most DNSBLs aren't accurate enough as outright
reject/blocks. SpamAssassin is much more accurate than MTA blacklists.
Daniel
--
Daniel Quinlan anti-spam (SpamAssassin), Linux,
http://www.pathname.com/~quinlan/ and open source consulting
Re: How to optimize SA?
Posted by Jason Haar <Ja...@trimble.co.nz>.
On Tue, Mar 09, 2004 at 02:51:54PM -0800, Justin Mason wrote:
> FWIW, though, SpamAssassin 2.6x has a very sophisticated DNSBL lookup
> algorithm which (a) runs all queries in parallel and (b) aborts ones
> that are taking significantly longer than the others, reducing problems
> if one or two hosts go down. So nowadays this should be *more* efficient
> run from SpamAssassin than from the MTA.
I don't doubt it. But that means spamc hangs around for up to 10 seconds
awaiting a response on a system that runs "spamd -m XX" - i.e. XX max
processes. If you ran RBL checks out on the SMTP server, then you may be
inefficient, but at least that 25K SMTP process is dealing with it rather
than a 28M spamd process...
--
Cheers
Jason Haar
Information Security Manager, Trimble Navigation Ltd.
Phone: +64 3 9635 377 Fax: +64 3 9635 417
PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1
Re: How to optimize SA?
Posted by Jason Haar <Ja...@trimble.co.nz>.
On Tue, Mar 09, 2004 at 03:41:48PM -0600, Steve Yuroff wrote:
> I've been running SA on a Cobalt Qube since 2.42 was current, and it's
> been a wonderful solution. Recently I moved to running spamd -d to
> because of the increased load the increase in spam was putting on the
> server. Now that's not cutting it- they're hitting me so hard I'm
> seeing processor loads in the 30's, and my poor little Qube is buckling
> under the load- I can't ssh in, email exchanges time out, ect.
First off, disable the network checks and move any RBL checks you relied on
SA for out onto your Internet-frontend SMTP server. That way you lose no
functionality, but remove a bunch of "hangs" from SA - which lead to
loading issues.
The DNS checks cannot be underestimated. We run Qmail-Scanner on all our
e-mails - and it's a perl script. It typically processes an e-mail in <0.3
secs, but calling spamc adds 2-8sec to it due to all the RBL lookups going
on.
[I prefer running them via SA and will do so until the load gets too much -
then I will take my own advice and move RBL back out to where it was
designed for ;-)]
--
Cheers
Jason Haar
Information Security Manager, Trimble Navigation Ltd.
Phone: +64 3 9635 377 Fax: +64 3 9635 417
PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1
Re: How to optimize SA?
Posted by Kelson Vibber <ke...@speed.net>.
At 01:56 PM 3/9/2004, Matt Kettler wrote:
>You can also gain speed, at a sacrifice of accuracy, by disabling network
>checks (razor, dnsbls, etc), bayes, and the AWL.
Another thing you can do, if you want to keep some network tests active, is
set low timeout values for the tests you do use.
Kelson Vibber
SpeedGate Communications <www.speed.net>
Re: How to optimize SA?
Posted by Matt Kettler <mk...@evi-inc.com>.
At 04:41 PM 3/9/2004, Steve Yuroff wrote:
>I've been running SA on a Cobalt Qube since 2.42 was current, and it's
>been a wonderful solution. Recently I moved to running spamd -d to
>because of the increased load the increase in spam was putting on the
>server. Now that's not cutting it- they're hitting me so hard I'm seeing
>processor loads in the 30's, and my poor little Qube is buckling under the
>load- I can't ssh in, email exchanges time out, ect.
>
>What else can I do to make SA run as quickly as possible- besides the
>obvious "buy a bigger server"?
First, you said you're running spamd -d... did you replace calls to
spamassassin with spamc? Otherwise using spamd doesn't help at all.
You can limit the load by using the -m parameter to spamd, forcing it to
limit the number of children, but then you need to use -f on spamc,
resulting in unfiltered mail.
You can also gain speed, at a sacrifice of accuracy, by disabling network
checks (razor, dnsbls, etc), bayes, and the AWL. Also ejecting some add-on
rulesets helps. A good first-check to find any culprits is to run a message
through spamassassin -D and see if it's getting "hung up" anywhere along
the way. If you're not on a current version (2.6x) you could be suffering
from a dead DNSBL.
Another method of dealing with it all is to limit the number of emails fed
to SA in the first place.. This usually requires an integration tool that
does queuing, like MailScanner. Of course, with this kind of system you
wind up trading off CPU load for increased queue depth and delayed mail
delivery. However, if you mail load is "bursty" and not continuous over the
whole day, this will deal with the spikes pretty nicely.