[Spamassassin Wiki] Update of "CorpusCleaning" by JustinMason

[Apache Wiki <wi...@apache.org>]

Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Spamassassin Wiki" for change notification.

The following page has been changed by JustinMason:
http://wiki.apache.org/spamassassin/CorpusCleaning

The comment on the change is:
a writeup of how to clean a corpus quickly

New page:
= Cleaning a corpus of FPs and FNs =

Here's how to clean a corpus of FalsePositives and FalseNegatives.

Firstly, do a mass-check.  You will wind up with a 'spam.log' and 'ham.log' file.  Run these commands to get a list of the 200 lowest-scoring spams, create a mbox file with just those messages, then open that mbox up in the "mutt" mail client:

{{{
sort -n +1 spam.log | head -200 > id.low
./mboxget < id.low > mbox
mutt -f mbox
}}}

(you could use another mail client if you want, it's just a std UNIX-format mbox file.)

Now, delete all messages that '''really are''' spams, and not false positives (or bounces, or virus blowback, or other kinds of undesirable messages).  Quit and save the mbox.  It now contains only the 'bad' messages.

You can then take that mbox file, grep out the original MassCheck message id strings, and remove those lines from the 'spam.log' file:

{{{
grep X-Mass-Check-Id mbox | sed -e 's/^X-Mass-Check-Id: //' > id.fps
./remove-ids-from-mclog id.fps < spam.log > spam.log.new
mv spam.log.new spam.log
}}}

You can also remove the offending files, or messages from the source mailboxes, directly.  However, this depends on what format you use to store messages; Maildirs, mboxes, etc. etc.  (Maildirs are easiest, since you can just delete the files named in the 'id.fps' file.)

Doing the same operation for FalseNegatives is similar, but reverses a few things... here's the commands to do that:

{{{
sort -rn +1 ham.log | head -200 > id.hi
./mboxget < id.hi > mbox
mutt -f mbox
}}}

Delete the messages that are good, usable ham, leaving only spams, virus blowback, bounces, or whatever other undesirable messages you want to get rid of.  Quit and save.

{{{
grep X-Mass-Check-Id mbox | sed -e 's/^X-Mass-Check-Id: //' > id.fns
./remove-ids-from-mclog id.fns < ham.log > ham.log.new
mv ham.log.new ham.log
}}}

Repeat, if necessary...