You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Brent Kennedy <br...@cfl.rr.com> on 2006/05/03 18:53:57 UTC
Silly Question
I am trying my best to explain to a non-technical person how spamassassin
works. The other issue I have is that I am trying to explain to them why
they need to drag mail to a public folder so the server can learn it as
spam. They say they are doing it but they get the message several times a
day.. Again.. I need to explain in laymans terms why they might get the same
message again a few more times until the server learns it.
My best layman explanations are below, please critique them, as I am having
trouble making a layman understand.
Here is my explanation for how spamassassin works(laymans terms):
1. Email comes into the mail server from the internet
2. Once the server has received the email, it then scans the email and
compares it to a list of rules. Every time a rule is matched, the server
adds points to the email.
3. When the server completes the scan of the email, it takes the total
number of points assigned to that email and compares that point value with
the maximum allowed points. If the point value is over the maximum the
server marks the message as spam.
4. if the server marks the email as spam, the email is forwarded to the
quarantine mailbox, if its marked as spam, the email is forwarded to the
intended recipient.
Here is my explanation for how spamassassin learns email as spam(laymans
terms):
1. Users receive the junk email
2. The users who received the junk email drags and drops the email to the
spammail public folder in outlook
3. Spamassassin connects to the internal email server and downloads the
email from the spammail public folder.
4. After the email is downloaded, the mail is scanned, the different aspects
of the email are noted and point rules inside the spamassassin engine are
updated(raised) based on the number of instances a rule is found in emails
from the spammail public folder.
Any help would be appreciated(thanks).
-Brent
Re: Silly Question
Posted by Kris Deugau <kd...@vianet.ca>.
Brent Kennedy wrote:
> I am trying my best to explain to a non-technical person how spamassassin
> works. The other issue I have is that I am trying to explain to them why
> they need to drag mail to a public folder so the server can learn it as
> spam. They say they are doing it but they get the message several times a
> day.. Again.. I need to explain in laymans terms why they might get the same
> message again a few more times until the server learns it.
>
> My best layman explanations are below, please critique them, as I am having
> trouble making a layman understand.
[snip]
I'd say you've got things about as simple as you can get short of
baby-talk. About all you could really do is think up another 3 or 4
ways to say the same things. <g>
With respect to the repeat missed spams, just say that the spammer is
sending multiple copies of the message, each one slightly different in a
way that's not obvious to the eye (it's not even a lie; message headers
will differ between two otherwise identical messages sent in the same
second - the MTA queue ID should be different). Over time,
SpamAssassin's learning system will pick up on the common pieces and
place less "weight" on the (hidden) differing ones.
-kgd
Re: Silly Question
Posted by jp <jp...@saucer.midcoast.com>.
I don't do the learning thing.
http://www.midcoast.com/help/email/spam.html is how we explain it to
users.
On Wed, May 03, 2006 at 12:53:57PM -0400, Brent Kennedy wrote:
> I am trying my best to explain to a non-technical person how spamassassin
> works. The other issue I have is that I am trying to explain to them why
> they need to drag mail to a public folder so the server can learn it as
> spam. They say they are doing it but they get the message several times a
> day.. Again.. I need to explain in laymans terms why they might get the same
> message again a few more times until the server learns it.
>
> My best layman explanations are below, please critique them, as I am having
> trouble making a layman understand.
>
>
> Here is my explanation for how spamassassin works(laymans terms):
>
> 1. Email comes into the mail server from the internet
> 2. Once the server has received the email, it then scans the email and
> compares it to a list of rules. Every time a rule is matched, the server
> adds points to the email.
> 3. When the server completes the scan of the email, it takes the total
> number of points assigned to that email and compares that point value with
> the maximum allowed points. If the point value is over the maximum the
> server marks the message as spam.
> 4. if the server marks the email as spam, the email is forwarded to the
> quarantine mailbox, if its marked as spam, the email is forwarded to the
> intended recipient.
>
>
> Here is my explanation for how spamassassin learns email as spam(laymans
> terms):
>
> 1. Users receive the junk email
> 2. The users who received the junk email drags and drops the email to the
> spammail public folder in outlook
> 3. Spamassassin connects to the internal email server and downloads the
> email from the spammail public folder.
> 4. After the email is downloaded, the mail is scanned, the different aspects
> of the email are noted and point rules inside the spamassassin engine are
> updated(raised) based on the number of instances a rule is found in emails
> from the spammail public folder.
>
>
> Any help would be appreciated(thanks).
>
> -Brent
--
/*
Jason Philbrook | Midcoast Internet Solutions - Internet Access,
KB1IOJ | Hosting, and TCP-IP Networks for Midcoast Maine
http://f64.nu/ | http://www.midcoast.com/
*/
Re: Silly Question
Posted by Mike Jackson <mj...@mightymerchant.com>.
> 2. Once the server has received the email, it then scans the email and
> compares it to a list of rules. Every time a rule is matched, the server
> adds points to the email.
> 3. When the server completes the scan of the email, it takes the total
> number of points assigned to that email and compares that point value with
> the maximum allowed points. If the point value is over the maximum the
> server marks the message as spam.
I typically expand these points when I'm explaining it to my users. I say
something along the lines of...
SA compares the message to characteristics of spam messages that other
people have seen. Each characteristic is a rule, and each rule has a point
value for how strong an indicator of spam it is. When a rule is matched, its
point value is added to the message's total. If the final total is higher
than a certain threshold, the message is considered spam.
You could also explain that there's some rules with negative point values,
and these represent characteristics that are more common of legitimate
messages, but that's an added detail that only muddies the waters.
And FWIW, I don't think it's a silly question at all. Some people have the
luxury of only filtering their own mail, and can be as hard-line as they
want. The rest of us have to work with laymen who have no idea how email
really works and are simply frustrated that they receive junk.
RE: Silly Question
Posted by Brent Kennedy <br...@cfl.rr.com>.
Thanks :)
And Knowing is half the battle(queue gi joe music)
Seriously.. I had not really had time to open the hood and look inside..
Thanks for the in-depth explanation.
Someone should put that up on the wiki site.
-Brent
-----Original Message-----
From: Matt Kettler [mailto:mkettler@evi-inc.com]
Sent: Wednesday, May 03, 2006 4:06 PM
To: Brent Kennedy
Cc: users@spamassassin.apache.org
Subject: Re: Silly Question
Brent Kennedy wrote:
> Wow.. You definitely went the opposite direction. Although, I did
> appreciate a well written explanation of the bayes system.
Well, I intentionally went the opposite way, to make sure that at least you
realize what is really going on. I wasn't really intending that to be
targeted at your users, I was targeting it to you.. :)
>
> I could be evil and forward this to them(thoughts?)... Maybe they wont
> ask again. >:)
I would reserve that for those who insist on questioning further.
"Ok, fine. I simplified a lot. Here's the gory details..."
Re: Silly Question
Posted by Matt Kettler <mk...@evi-inc.com>.
Brent Kennedy wrote:
> Wow.. You definitely went the opposite direction. Although, I did
> appreciate a well written explanation of the bayes system.
Well, I intentionally went the opposite way, to make sure that at least you
realize what is really going on. I wasn't really intending that to be targeted
at your users, I was targeting it to you.. :)
>
> I could be evil and forward this to them(thoughts?)... Maybe they wont ask
> again. >:)
I would reserve that for those who insist on questioning further.
"Ok, fine. I simplified a lot. Here's the gory details..."
RE: Silly Question
Posted by Brent Kennedy <br...@cfl.rr.com>.
Wow.. You definitely went the opposite direction. Although, I did
appreciate a well written explanation of the bayes system.
I could be evil and forward this to them(thoughts?)... Maybe they wont ask
again. >:)
-Brent
-----Original Message-----
From: Matt Kettler [mailto:mkettler@evi-inc.com]
Sent: Wednesday, May 03, 2006 1:47 PM
To: Brent Kennedy
Cc: users@spamassassin.apache.org
Subject: Re: Silly Question
Brent Kennedy wrote:
> Here is my explanation for how spamassassin learns email as
> spam(laymans
> terms):
>
> 1. Users receive the junk email
> 2. The users who received the junk email drags and drops the email to
> the spammail public folder in outlook 3. Spamassassin connects to the
> internal email server and downloads the email from the spammail public
> folder.
> 4. After the email is downloaded, the mail is scanned, the different
> aspects of the email are noted and point rules inside the spamassassin
> engine are
> updated(raised) based on the number of instances a rule is found in
> emails from the spammail public folder.
Statement 4 is untrue, but I can understand the reason to over-simplify
here.
The real story is a bit complex and long-winded. However, it is important
that you at least know that learning spam does not adjust the score of any
of rule in SA. The whole sa-learn process does not ever even consider any of
the rules in SA at all, it is completely unaware that they exist.
The learning only affects what BAYES category the message will be placed in.
It does not adjust scores, but it does indirectly affect the score by
shifting which BAYES_* rule will later match.
Let me take a moment to explain bayes. Bayes is one of the many subsystems
contained in SpamAssassin. While it is quite complex if you get down to the
fine-grained details, it's not that hard at a high-level view. As a simple
explanation, it's a statistical "probability of spam" tracker for all the
words found in the body of messages. It establishes spam probabilities for
words through training, and later applies these to compute overall chance of
spam for a message when you scan.
In greater detail:
Fundamentally, bayes works by breaking a message into tokens. These tokens
are mostly just the words in the body of the message, but in SA's bayes
implementations there are also tokens extracted from some parts of the
message headers and URLs.
When you are training messages with sa-learn, it records how many times each
token is seen in a spam message and how many times in nonspam. These
statistics are stored in a database for future reference. So when you train
a message as spam, sa-learn breaks the message into tokens, finds all the
ones present in the database and increments their spam counter. All the
other tokens get added as new tokens with the spam counter set to 1 and the
nonspam counter set to 0.
Based on the frequency of spam vs nonspam, SA can compute that a token has a
particular probability of being present in a spam message vs a nonspam
message.
When SA uses bayes to scan a message, it looks up all the tokens present in
the database, computes all the probabilities of each token, then combines
them to get an overall probability that the whole message is spam or not.
This is expressed as a percentage chance of the message being spam (from 0%
to 100%)
So through your training, SA's bayes effectively learns to associate certain
words with spam, some with nonspam, and some as being "in between". When a
message is scanned let's say for example it has 5 words that are associated
strongly with spam, 10 that are "in between" and 1 that's strongly
associated with nonspam. Since there are so many more strong spam tokens
than non-spam tokens, bayes is going to declare this message to have a
fairly high probability of being spam.
These bayes probabilities show up as fixed number rules that SA assigns
scores to. These scores are pre-generated, like every other score in SA, and
do not change unless you perform a sa-update or otherwise upgrade your SA to
have new rules.
In SA 3.1.x the rules that bayes shows up as are BAYES_00 (0%-1%), BAYES_05
(1%-5%), BAYES_20 (5%-20%), BAYES_40 (20%-40%), BAYES_50 (40%-60%), BAYES_60
(60%-80%), BAYES_80 (80%-95%) BAYES_95 (95%-99%) and BAYES_99 (99%-100%).
By shifting which of these rules a message matches, sa-learn winds up
affecting the overall score of future messages with similar content.
Re: Silly Question
Posted by Matt Kettler <mk...@evi-inc.com>.
Brent Kennedy wrote:
> Here is my explanation for how spamassassin learns email as spam(laymans
> terms):
>
> 1. Users receive the junk email
> 2. The users who received the junk email drags and drops the email to the
> spammail public folder in outlook
> 3. Spamassassin connects to the internal email server and downloads the
> email from the spammail public folder.
> 4. After the email is downloaded, the mail is scanned, the different aspects
> of the email are noted and point rules inside the spamassassin engine are
> updated(raised) based on the number of instances a rule is found in emails
> from the spammail public folder.
Statement 4 is untrue, but I can understand the reason to over-simplify here.
The real story is a bit complex and long-winded. However, it is important that
you at least know that learning spam does not adjust the score of any of rule in
SA. The whole sa-learn process does not ever even consider any of the rules in
SA at all, it is completely unaware that they exist.
The learning only affects what BAYES category the message will be placed in. It
does not adjust scores, but it does indirectly affect the score by shifting
which BAYES_* rule will later match.
Let me take a moment to explain bayes. Bayes is one of the many subsystems
contained in SpamAssassin. While it is quite complex if you get down to the
fine-grained details, it's not that hard at a high-level view. As a simple
explanation, it's a statistical "probability of spam" tracker for all the words
found in the body of messages. It establishes spam probabilities for words
through training, and later applies these to compute overall chance of spam for
a message when you scan.
In greater detail:
Fundamentally, bayes works by breaking a message into tokens. These tokens are
mostly just the words in the body of the message, but in SA's bayes
implementations there are also tokens extracted from some parts of the message
headers and URLs.
When you are training messages with sa-learn, it records how many times each
token is seen in a spam message and how many times in nonspam. These statistics
are stored in a database for future reference. So when you train a message as
spam, sa-learn breaks the message into tokens, finds all the ones present in the
database and increments their spam counter. All the other tokens get added as
new tokens with the spam counter set to 1 and the nonspam counter set to 0.
Based on the frequency of spam vs nonspam, SA can compute that a token has a
particular probability of being present in a spam message vs a nonspam message.
When SA uses bayes to scan a message, it looks up all the tokens present in the
database, computes all the probabilities of each token, then combines them to
get an overall probability that the whole message is spam or not. This is
expressed as a percentage chance of the message being spam (from 0% to 100%)
So through your training, SA's bayes effectively learns to associate certain
words with spam, some with nonspam, and some as being "in between". When a
message is scanned let's say for example it has 5 words that are associated
strongly with spam, 10 that are "in between" and 1 that's strongly associated
with nonspam. Since there are so many more strong spam tokens than non-spam
tokens, bayes is going to declare this message to have a fairly high probability
of being spam.
These bayes probabilities show up as fixed number rules that SA assigns scores
to. These scores are pre-generated, like every other score in SA, and do not
change unless you perform a sa-update or otherwise upgrade your SA to have new
rules.
In SA 3.1.x the rules that bayes shows up as are BAYES_00 (0%-1%), BAYES_05
(1%-5%), BAYES_20 (5%-20%), BAYES_40 (20%-40%), BAYES_50 (40%-60%), BAYES_60
(60%-80%), BAYES_80 (80%-95%) BAYES_95 (95%-99%) and BAYES_99 (99%-100%).
By shifting which of these rules a message matches, sa-learn winds up affecting
the overall score of future messages with similar content.
RE: Silly Question
Posted by David B Funk <db...@engineering.uiowa.edu>.
On Wed, 3 May 2006, Brent Kennedy wrote:
> rules. The issue before here, is that spamassassin doesn't split emails up
> between recipients when a message is sent to multiple people. If one user
> is on the whitelist_to or all_spam_to or some_spam_to list, then everyone
> gets its.
[snip..]
This is TOTALLY dependent upon how you've integrated SA into your mail
system.
If you filter at delivery time (procmail, etc) then you're filtering on
a per-user basis (more CPU overhead, cannot do SMTP-reject, but can do
per-user customizations).
If you filter at the MTA level (sendmail milter, procmail pipe, etc)
then you're doing per-message filtering and cannot easily do per-user
customizations. It is possible to create a MTA system that splits up
messages so to enable per-user customizations but it is either fragile
or can get real complex/ugly.
Stupidly simple example, set your incoming MTA config so that it will
accept at most one recipient per message (EG for sendmail set
MaxRecipientsPerMessage=1). This will force the remote sending MTA to
split the message for you. Easy but fragile, not all MTAs will do
this successfully.
More complex, local layered MTAs, accept the message then split
internally (but if you want to be able to do SMTP-rejects based
upon spam score, it needs to get more complex/ugly ;(.
--
Dave Funk University of Iowa
<dbfunk (at) engineering.uiowa.edu> College of Engineering
319/335-5751 FAX: 319/384-0549 1256 Seamans Center
Sys_admin/Postmaster/cell_admin Iowa City, IA 52242-1527
#include <std_disclaimer.h>
Better is not better, 'standard' is better. B{
RE: Silly Question
Posted by Brent Kennedy <br...@cfl.rr.com>.
Unfortunately I have a user that is being insistent that they somehow be
involved since they are the victim of the spam messages.
I give them the drag and drop folder so that those of them who feel that we
are not doing anything about it can participate and hopefully make
themselves think they contributed to the effort and that their effort
somehow made a difference.
I do use Razor, DCC, RBL's, custom filters, a black list and some custom
rules. The issue before here, is that spamassassin doesn't split emails up
between recipients when a message is sent to multiple people. If one user
is on the whitelist_to or all_spam_to or some_spam_to list, then everyone
gets its. I was finally able to convince them to allow me to remove
everyone from the lists. I hope that reduces the spam.. In fact I have seen
a rize in marked messages.. we are at 85% spam now that the whitelists were
cleared.
You did remind me to correct a PTR issue with brighthouse/road runner for
our domain. Anyone else have a problem with getting them to do a reverse
PTR?
-Brent
-----Original Message-----
From: Greg Allen [mailto:salist@floridacpu.com]
Sent: Wednesday, May 03, 2006 1:32 PM
To: Brent Kennedy; users@spamassassin.apache.org
Subject: RE: Silly Question
I know some people use the public folder drag-drop for learning spam, but I
personally don't like the whole idea. And I did consider it.
I would rather work smarter on the server end to kill or mark the spam
before it gets to the user.
I don't think users should have to worry about all the technical details,
that's my job.
Yea, a few will slip through now and then. But they will slip through either
way, and annoying users with the details isn't going to change that.
Spamassassin is pretty smart. If you enable all of the features and keep the
version up-to-date, tweak a little and configure your server (Postfix for
instance) to do better at rejecting UCE up-front, most spams will eventually
classify themselves correctly automatically.
The way they classify themselves is because the spammer will use various
email servers and will start getting on various IP blacklists and URBL
lists, various headers are seen, etc. Within a few days the new spam will
start to have more and more points. All it takes is to trip the spam points
one time and that's is the end of that.
If you use Razor, DCC, URBL, etc... system admins (and users in some systems
like DCC, etc) are constantly feeding spams into those systems. That is a
better way to go IMO. use a system that is already setup for that.
For the marginal (hard to define) spam emails that go on and on for weeks,
they can forward those emails to the admin to find a way to block.
You will always have a certain amount of white-list black-list admin needing
to be done.
Also, now that AOL is blocking all email with no PTR record, you can
probably kill a lot of spam with that right on the front end now. If anyone
complains, ask them how they email to AOL. :-)
> -----Original Message-----
> From: Brent Kennedy [mailto:brentk@cfl.rr.com]
> Sent: Wednesday, May 03, 2006 12:54 PM
> To: users@spamassassin.apache.org
> Subject: Silly Question
>
>
> I am trying my best to explain to a non-technical person how
> spamassassin works. The other issue I have is that I am trying to
> explain to them why they need to drag mail to a public folder so the
> server can learn it as spam. They say they are doing it but they get
> the message several times a day.. Again.. I need to explain in laymans
> terms why they might get the same message again a few more times until
> the server learns it.
>
> My best layman explanations are below, please critique them, as I am
> having trouble making a layman understand.
>
>
> Here is my explanation for how spamassassin works(laymans terms):
>
> 1. Email comes into the mail server from the internet 2. Once the
> server has received the email, it then scans the email and compares it
> to a list of rules. Every time a rule is matched, the server adds
> points to the email.
> 3. When the server completes the scan of the email, it takes the total
> number of points assigned to that email and compares that point value
> with the maximum allowed points. If the point value is over the
> maximum the server marks the message as spam.
> 4. if the server marks the email as spam, the email is forwarded to
> the quarantine mailbox, if its marked as spam, the email is forwarded
> to the intended recipient.
>
>
> Here is my explanation for how spamassassin learns email as
> spam(laymans
> terms):
>
> 1. Users receive the junk email
> 2. The users who received the junk email drags and drops the email to
> the spammail public folder in outlook 3. Spamassassin connects to the
> internal email server and downloads the email from the spammail public
> folder.
> 4. After the email is downloaded, the mail is scanned, the different
> aspects of the email are noted and point rules inside the spamassassin
> engine are
> updated(raised) based on the number of instances a rule is found in
> emails from the spammail public folder.
>
>
> Any help would be appreciated(thanks).
>
> -Brent
>
>
>
RE: Silly Question
Posted by Greg Allen <sa...@floridacpu.com>.
I know some people use the public folder drag-drop for learning spam, but I
personally don't like the whole idea. And I did consider it.
I would rather work smarter on the server end to kill or mark the spam
before it gets to the user.
I don't think users should have to worry about all the technical details,
that's my job.
Yea, a few will slip through now and then. But they will slip through either
way, and annoying users with the details isn't going to change that.
Spamassassin is pretty smart. If you enable all of the features and keep the
version up-to-date, tweak a little and configure your server (Postfix for
instance) to do better at rejecting UCE up-front, most spams will eventually
classify themselves correctly automatically.
The way they classify themselves is because the spammer will use various
email servers and will start getting on various IP blacklists and URBL
lists, various headers are seen, etc. Within a few days the new spam will
start to have more and more points. All it takes is to trip the spam points
one time and that's is the end of that.
If you use Razor, DCC, URBL, etc... system admins (and users in some systems
like DCC, etc) are constantly feeding spams into those systems. That is a
better way to go IMO. use a system that is already setup for that.
For the marginal (hard to define) spam emails that go on and on for weeks,
they can forward those emails to the admin to find a way to block.
You will always have a certain amount of white-list black-list admin needing
to be done.
Also, now that AOL is blocking all email with no PTR record, you can
probably kill a lot of spam with that right on the front end now. If anyone
complains, ask them how they email to AOL. :-)
> -----Original Message-----
> From: Brent Kennedy [mailto:brentk@cfl.rr.com]
> Sent: Wednesday, May 03, 2006 12:54 PM
> To: users@spamassassin.apache.org
> Subject: Silly Question
>
>
> I am trying my best to explain to a non-technical person how spamassassin
> works. The other issue I have is that I am trying to explain to them why
> they need to drag mail to a public folder so the server can learn it as
> spam. They say they are doing it but they get the message several times a
> day.. Again.. I need to explain in laymans terms why they might
> get the same
> message again a few more times until the server learns it.
>
> My best layman explanations are below, please critique them, as I
> am having
> trouble making a layman understand.
>
>
> Here is my explanation for how spamassassin works(laymans terms):
>
> 1. Email comes into the mail server from the internet
> 2. Once the server has received the email, it then scans the email and
> compares it to a list of rules. Every time a rule is matched, the server
> adds points to the email.
> 3. When the server completes the scan of the email, it takes the total
> number of points assigned to that email and compares that point value with
> the maximum allowed points. If the point value is over the maximum the
> server marks the message as spam.
> 4. if the server marks the email as spam, the email is forwarded to the
> quarantine mailbox, if its marked as spam, the email is forwarded to the
> intended recipient.
>
>
> Here is my explanation for how spamassassin learns email as spam(laymans
> terms):
>
> 1. Users receive the junk email
> 2. The users who received the junk email drags and drops the email to the
> spammail public folder in outlook
> 3. Spamassassin connects to the internal email server and downloads the
> email from the spammail public folder.
> 4. After the email is downloaded, the mail is scanned, the
> different aspects
> of the email are noted and point rules inside the spamassassin engine are
> updated(raised) based on the number of instances a rule is found in emails
> from the spammail public folder.
>
>
> Any help would be appreciated(thanks).
>
> -Brent
>
>
>