Spamassassin for TREC (fwd)

[jm...@jmason.org]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


bringing this onto dev -- a bit of correspondence with Gordon Cormack re
TREC.

Gord, the main thing I'd be worried about with those scores, is that it
wouldn't be testing SpamAssassin "out of the box".  I get your point about
boosting the BAYES scores, but the other rules, in particular
MIME_HTML_ONLY?  not so sure about that.  in our corpora, 4.17% of ham
hits that -- which is really quite a lot.

- --j.

- ------- Forwarded Message
Date:    Mon, 29 Aug 2005 08:42:48 -0400
> From:    "Gordon V. Cormack" <gv...@uwaterloo.ca>
To:      jm@jmason.org (Justin Mason)
Subject: Spamassassin for TREC

Justin,

Nobody from Spamassassin signed up for TREC.  Nevertheless, I would
like to include a Spamassassin run in the results.

Here is the latest evaluation toolkit wrapper that we've published.
I'm happy to go with this (note that as a side-effect I can extract
bayes-only and no-bayes result files).

But if you, or somebody else, would like to suggest changes, I'm
all ears.

Note that this setup is very similar to "sa-supervised" in my
paper, but with Spamassassin 3.



- ------- Forwarded Message 2
From: jm@jmason.org (Justin Mason)
Date: Wed, 07 Sep 2005 22:05:06 -0700
To: "Gordon V. Cormack" <gv...@uwaterloo.ca>
Cc: jm@jmason.org (Justin Mason)
Subject: Re: Spamassassin for TREC


Gordon V. Cormack writes:
> Nobody from Spamassassin signed up for TREC.  Nevertheless, I would
> like to include a Spamassassin run in the results.
> 
> Here is the latest evaluation toolkit wrapper that we've published.
> I'm happy to go with this (note that as a side-effect I can extract
> bayes-only and no-bayes result files).
> 
> But if you, or somebody else, would like to suggest changes, I'm
> all ears.
> 
> Note that this setup is very similar to "sa-supervised" in my
> paper, but with Spamassassin 3.

Hey Gord!

sorry, we dropped the ball on signing up for TREC!   Sorry for the delay,
in replying, too, I was away on vacation last week.  

one bit of feedback -- I'm not sure why the score changes are added in
'filters-new/spamassassin/initialize':

  echo score MIME_HTML_ONLY 3 >> .spamassassin/user_prefs
  echo score BAYES_60 2 >> .spamassassin/user_prefs
  echo score BAYES_70 2.6 >> .spamassassin/user_prefs
  echo score BAYES_80 4.2 >> .spamassassin/user_prefs
  echo score BAYES_90 4.3 >> .spamassassin/user_prefs
  echo score BAYES_99 5 >> .spamassassin/user_prefs
  echo score HABEAS_SWE 0 0 0 0 >> .spamassassin/user_prefs
  echo score MICROSOFT_EXECUTABLE 4 >> .spamassassin/user_prefs
  echo score RCVD_IN_DSBL 4.3 >> .spamassassin/user_prefs
  echo score RCVD_IN_SBL 1 >> .spamassassin/user_prefs

all of those would appear to be superfluous, in my opinion.

Also, does the current working dir when
'filters-new/spamassassin/initialize' is called, match $HOME?  Because if
not, '.spamassassin/user_prefs' may not be used, if it's not the same file
as "~/.spamassassin/user_prefs".

Using the "-p" switch when calling the various sa-learn and spamassassin
commands could make an effective workaround for this btw.

Apart from those issues, it looks entirely correct... let me know
if anything else is needed.

(BTW is it ok to forward our correspondence -- yr mail and this reply --
to a public list?  We prefer to operate publicly where possible, to
encourage the open source aspects of our operation.)

- --j.


- ------- Forwarded Message 3
From: "Gordon V. Cormack" <gv...@uwaterloo.ca>
Date: Thu, 8 Sep 2005 03:50:41 -0400 (00:50 PDT)
To: jm@jmason.org (Justin Mason)
Subject: Re: Spamassassin for TREC

Justin, 

Thanks for the response.  By all means post my message.

The scores that are used are the ones that I have found work
best  - they are the scores used in Mr X's setup (since 
before the corpus was created) and the ones I recommend:

   http://plg.uwaterloo.ca/~gvcormac/spamassassin

The most important one is BAYES_99.  By default the
bayesian filter doesn't have enough weight to veto
a message (using the default threshold of 5, that is).

Yes, the setup runs with $HOME set to `pwd`

cheers,

Gord


- -- 
Gordon V. Cormack     CS Dept, University of Waterloo, Canada N2L 3G1
gvcormack@uwaterloo.ca            http://cormack.uwaterloo.ca/cormack

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)
Comment: Exmh CVS

iD8DBQFDIOdwMJF5cimLx9ARAkKHAKCtId1QiXVcq5DkhK4tjJ5s51XMOACfc+Wt
J7qrasu/v9Wirv1ZVQga+qw=
=IVbk
-----END PGP SIGNATURE-----

Re: Spamassassin for TREC (fwd)

[Loren Wilton <lw...@earthlink.net>]

Note also

>   echo score MICROSOFT_EXECUTABLE 4 >> .spamassassin/user_prefs

Isn't that a 2.6x rule that went away in 3.0?  I would hope that anything
comparing filtering results (as I would guess this to be, knowing nothing of
it) would be using a reasonably recent version.

(Of course it would also be nice to see a run with reasonable SARE rules and
RDJ as an automatic maintainer.)

        Loren