Spam Assassin overview for UNIX accounts

[Padraig Houlahan <pj...@lowell.edu>]

I would like to submit my help page that I wrote for my users concerning spam 
assassin for possible use on the Spam Assassin documentation page. Much of the 
existing documentation is too obtuse for users and administrators who just 
want to get a configuration working quickly which is why I wrote this 
overview. I think this kind of overview might be very helpful for others.

Regards,

PH.



Topic: SPAM ASSASSIN OVERVIEW FOR UNIX ACCOUNTS


DISCUSSION:


Spam Assassin is a spam control package that
is available to you and is activated by creating
a file ('.spam.filter') in your UNIX home directory. 
SA is tuned by setting weightings to various characteristics
associated with spam in a global (site wide) configuration
file. As a result, when a piece of mail has an accumulated
a total weight or score that exceeds a certain threshold,
the subject line is modified to include the word "SPAM."

One of the really nice features of SA is that email identified
as spam has its contents stored as an attachment so you don't
have to view the content if you later review such email.

SA maintains a database and can learn from experience
and so is self tuning as it gets more example files to
work with. The process of tuning usually involves the
learning program (called sa-learn.) SA uses probabilistic
models to identify spam which is why training is essential
and results improve over time.

The documentation describing how SA works is very
confusing and hard to understand, however, it seems that
these are the main modes of operation:

a. Mail is tested against a set of rules that search
	for specific words and patterns. Each rule
	contributes to an overall score, and if the
	score exceeds a threshold, it is identified
	as spam and '***SPAM***' is added to the
	subject line. 	

b. If the user allows auto_learning, SA can learn
	from extreme scores how spam is evolving and
	adapt itself accordingly.

c. Users can actively help SA learn by training it 
	against files of spam and ham (good email.)
	This step only accelerates the process.
	It is not mandatory - learning can occur
	naturally.

d. Users can customize SA for their mail





BASIC USEAGE

Using SA is very easy - just create a file called
'.spam.filter' in your home directory, e.g.

	cd ~
	touch .spam.filter

>From this point on, most of your spam will be flagged 
as spam. Combined with a ruleset in your mail reader 
(Eudora or Outlook perhaps) that either deletes 
(or archives) any mail with ***SPAM***
in the subject header, you will find a huge
reduction in the volume of your spam.

It is very likely that most of our users will find this 
an acceptable solution and will not want the hassle 
of proceeding to the next level of useage - training SA.





TRAINING SPAM ASSASSIN

If you want to try and teach SA about your spam...

Because of the difficulty in defining spam for all users,
individuals can augment spam detection by doing the following:

	1. In your home directory, create
		a directory called .spamassassin
		and copy in a default set of
		files to work with - contact
		a system administrator for these.

	2. Train SA to recognize Ham and Spam (where
		'ham' is good email. Do this by:

		sa-learn --ham --mbox Mail/ham

		or

		sa-learn --spam --mbox Mail/spam

(You might need to use a full path, e.g. /opt/local/bin/sa-learn...)

When training, the --mbox argument tells sa-learn that the
file is in standard mail format.

Typically, training needs to involve hundreds of examples of
each kind (ham or spam) for SA to become effective.

None of this actually blocks the mail from appearing 
on your screen. However, if you use a Mac or PC that
allows you to set mail filters, you can set a filter
that automatically handles any email with the ***SPAM***
tag in the subject line to either delete it outright,
or to move it to a designated spam folder. This way the
spam never appears in your inbox.

It takes about 500 - 1000 examples of ham and spam to
properly train the system. You have to be careful how you
train - don't use a mailfile from 5yrs ago as ham
in the notion that little spam existed then, since the
training might just teach SA that old mail is ham
and new is spam!




CUSTOMIZING YOUR SETTINGS

Users can customize their filtering by modifying the
user_prefs file in the .spamassassin directory that resides
in their home directory.

SA has hundreds of rules that incoming mail is tested against. A
list of the current version rules can be seen at the SA website

	www.spamassassin.org/test.html

As incoming mail is tested against each rule, the cumulative score is
incremented (depending on what the rule thinks) and if the total
score exceed a certain threshold (5 by default) the mail is
tagged as spam.

To make make SA reject more spam, decrease the threshold by adding 
a line like:

required_hits		3

to your user_prefs file.

[My use of a threshold this low stems from my experience that
this will work for me with my particular combination of tuned
scores that I assign. If you want to be less aggressive, or
if you assign higher scores to various rulesets, you might need
to use a higher threshold.]

While SA adds header information to mail it identifies as spam, 
you can also have it add useful information to non-spam by 
adding a line to your user_prefs file like:

add_header all Status _YESNO_, hits=_HITS_, tests=_TESTS_

This will include what the hit/score was and what tests were triggered
in the non-spam email.  This is useful for identifying tests to be tuned
when mail is not properly identified as spam.

You can tune a test weighting or score contribution by adding
a line for each test like:

score EXCUSE_14                 .5

where, in this example, EXCUSE_14 is a ruleset in SA that detects 
phrases concerned with how to stop further spam, and 0.5 is the 
weighting you want to assign.

Again, useful tests to monitor can be seen in the Status header above.
For example, by examing the Status header, you can see what tests were 
triggered, and what the total hit score was. You can then decide if you
want to increase the weighting of some of these tests so similar
mail is identified as spam the next time.

Be careful about how you tune scores. Ideally, you want a combination of
tests to identify spam since then no one test can be accidently tripped
by non-spam, and it makes it harder to create spam that thwarts multiple
tests.




WHITELISTS and BLACKLISTS

You can guarantee that mail from a particular source gets though the
spam filter by adding it to a whitelist, e.g.

whitelist 	joe@aol.com

and you can block a site or user with a blacklist:

blacklist 	*.spamhaus.tw

Our site default whitelists give large negative scores to .edu, .mil, and
.gov sites to ensure their mail gets through. However, edu sites are
being spoofed more regularly and in particular, we get spam
supposedly originating from our edu domain. For this reason, I have
de-activated the edu "whitelist" by using

unwhitelist	*.edu

in my personal user_prefs file and letting .edu mail be tested like any
other email.





HOW GOOD IS SPAM ASSASSIN?


Since undertaking a major effort in to learn about SA and
keeing statistics, here is what I found after a month:

1496 - Total emails received
   3 - False positives
 604 - Correctly identified spam
  49 - Missed spam

As I have better tuned my user_prefs file, the last 210
of these emails resulted in no false positives and only 7 missed spam.




EXAMPLE USER_PREFS FILE


Here's a recent version of my user_prefs:


#++++++++++++++++++++++++++++++++++++++++++++++++++++++++
#
#
# You need to regularly check your spam folder
# for false positives!
#

required_hits		2.9


unwhitelist_from  *.edu
whitelist_from  *amazon.com
whitelist_from  *.gov
whitelist_from  *.mil


add_header all Status _YESNO_, hits=_HITS_, tests=_TESTSCORES_, required=_REQD_

score HEADER_8BITS		0.1
score HTML_COMMENT_8BITS	0.1
score HTML_MESSAGE		1.4
score HTML_10_20		1.5
score HTML_50_60		1.5
score HTML_60_70		1.5
score HTML_FONT_BIG		1.5
score SUBJ_FULL_OF_8BITS	0.1
score UPPERCASE_25_50		0.7
score UPPERCASE_40_50		0.7
score UPPERCASE_50_75		0.7
score UPPERCASE_75_100		0.7
score REMOVE_IN_QUOTES 		0.4
score CLICK_TO_REMOVE 		0.4
score LARGE_HEX 		0.4
score MIME_BASE64_ILLEGAL 	0.4
score MIME_BASE64_TEXT	 	0.4
score MIME_HTML_ONLY	 	1
score RCVD_IN_SBL		1.1
score RCVD_IN_NJABL		1
score RCVD_IN_RFCI		1.1
score RCVD_IN_UNCONFIRMED_DSBL	1.1
score EXCUSE_14			.5
score EXCUSE_3			.5
score EXCUSE_11			1
score NO_REAL_NAME		3.0
score MISSING_HEADERS		1.3
score OFFERS_ETC		1.0
score BAYES_20			-2.8
score BAYES_30			-1.8
score BAYES_40			-0.8
score BAYES_50			0.0
score BAYES_60			0.8
score BAYES_70			1.2
score BAYES_80			1.4
score BAYES_90			2.4
score PORN_4			2.0