Update to the Pyzor plugin (Re: bug 6108)

[James Birkett <ja...@cyberhound.com>]

Hi,

Bug 6108 says is that the pyzor plugin ignores whitelisting entirely, and a
comment on that bug suggests updating the Pyzor plugin to use Wilson Score
formula described here
http://www.evanmiller.org/how-not-to-sort-by-average-rating.html

I have tested this using a spreadsheet and our own corpus of spam and ham
email and got good results, so I went ahead and implemented it in the Pyzor
plugin. I'm new to both spamassassin and perl so sorry if it's a bit rough
around the edges.

I wanted to make it possible to have multiple spamassassin rules with
different wilson score parameters giving different spamassassin scores, but
obviously we don't want to query pyzor multiple times on the same mail so
I've changed the actual pyzor lookup to be done during
extract_metadata() instead
of when the eval-rule is run. It works, but it looks to me like the query
to pyzor will block while waiting for a response and I don't know the
spamassassin well enough to know if this will cause problems.  Please could
someone who knows the internals better take a look and give me some
feedback?

Thanks very much,

PS: this is based off the pyzor plugin from version 3.4.0 because that's
what we're using at the moment (I know, it's old, sorry...). I've not
looked at more recent versions but if there is interest in including this
work upstream I'd be happy to rebase my changes off of SVN.

-- 
James Birkett

Scanned by CyberHound
(http://cyberhound.com/)


Confidentiality Notice: This email, including any attachments, is confidential and may be privileged.  If you are not the intended recipient please notify the sender immediately and delete it. You should not copy it or use it for any purpose or disclose its contents to any other person without CyberHound's prior written permission. CyberHound Pty Ltd reserves the right to monitor all email communications passing through its networks and devices.

Re: Update to the Pyzor plugin (Re: bug 6108)

[James Birkett <ja...@cyberhound.com>]

On 11 March 2017 at 03:29, RW <rw...@googlemail.com> wrote:

> On Sun, 26 Feb 2017 19:35:11 +1000
> James Birkett wrote:
>
> > Hi,
> >
> > Bug 6108 says is that the pyzor plugin ignores whitelisting entirely,
> > and a comment on that bug suggests updating the Pyzor plugin to use
> > Wilson Score formula described here
> > http://www.evanmiller.org/how-not-to-sort-by-average-rating.html
>
> SpamAssassin allows easy reporting of whole mailboxes of spam, but to
> report ham you have to go to the website and type in captchas (I think
> there are a few trusted admins that can bypass this). There's liable to
> be a substantial bias in reporting, and the bias is likely to vary
> strongly from hash to hash. I don't know for sure, but I suspect that a
> lot of whitelisting counts may be historic.
>
> Emails where the body contains only links or addresses get
> predigested down to a null string and all have the same hash with a
> 26.6:1 reporting ratio. There are numerous variants with additional
> boilerplate text. To eliminate this class of FP you need a threshold of
> at least 0.97.
>

Yep, spotted those in both spam and ham. I've used a pyzor local whitelist
on some of the most common hashes. The empty string, a few variants on the
theme of "sent from my iPhone", as well as some Google calendar emails.


>
> All in all I don't think it makes sense to use anything other than very
> high thresholds. If you want to persue this I would suggest using a
> threshold of at least 0.99 and varying the z parameter instead.
>
>
> > I have tested this using a spreadsheet and our own corpus of spam and
> > ham email and got good results,
>
> I think a lot of the benefit you are seeing comes from the fact that
> you are rescanning corpus mail rather than using the pyzor counts at
> delivery time.
>

That definitely could be related, but a large number of these mails were
hitting pyzor and nothing else, and others were hitting pyzor and
low-scoring rules that weren't enough to push it over the threshold. A lot
of the examples were sexually-explicit spam, and it seems likely that the
Bayesian filter might get these mails but I'm running spam filtering for
customers and am not really able to train the Bayesian filter properly so I
have it disabled.


> Using Wilson doesn't just affect whitelisted emails it also sets the
> minimum number of reports needed for a pyzor hit. For your three rules
> that rises from 5 to 381, 22 and 12 (assuming that the very low z value
> in the first rule is an error). In production you will lose pyzor hits,
> and for new hashes, the most valuable early hits.
>
>
The low z value is not an error. It's the result of playing around with the
parameters in a spreadsheet to see what works best. It came as a surprise
to me too, but it does make some kind of sense - with a very high threshold
(e.g. 99.5%) and low z-score, you're essentially saying "There is at least
some chance that 99.5% of people would think this is spam". If there's no
whitelisting, 4 pyzor reports are enough to trigger it. With the setup I
described, personally I'm confident enough to short-circuit on pyzor with
Wilson parameters 99.5% threshold and 10% confidence (z=~0.126), so it
gives us a big boost.


> I think pyzor's own defaults are sensible - just ignore anything
> that's whitelisted.


Well I've seen very spammy spam that has been whitelisted, so I think the
wilson method has its advantages. I'm not able to share the corpus I used,
but I've attached the spreadsheet I used for the analysis. I couldn't find
all of the original messages I used but for the ones I could find I've put
the pyzor predigests in a zip file if you're interested. There's a cluster
of them with very sexually explicit messages that each have been
whitelisted once and listed between 600 and 700 times, the wilson method
would get these, but simply ignoring anything that is whitelisted would not.

> I wanted to make it possible to have multiple spamassassin rules with
> > different wilson score parameters giving different spamassassin
> > scores, but obviously we don't want to query pyzor multiple times on
> > the same mail so I've changed the actual pyzor lookup to be done
> > during extract_metadata() instead
> > of when the eval-rule is run.
>
> The usual way this would be done is to have both functions look for
> cached counts and run pyzor if they aren't there - see the BAYES_*
> rules for an example.
>

Oh.... For some reason I had assumed that the eval rules were eval'ed
concurrently in some kind of asynchronous way (possibly because I had
previously been looking at the DNSBL and URIBL code which does to use
callbacks), but that doesn't seem to be the case here, right? If you don't
have to worry about callbacks or threads then doing what you describe is
much easier than I had thought.

I'll give it a try and get back to you. I'll also add methods to match any
mail that has at least 'n' (typically 5) listings or at least 'n'
(typically 1) whitelistsings so you can use a meta rule to obtain the
standard pyzor behaviour.

Thanks very much,

-- 
James Birkett
Cyberhound

Scanned by CyberHound
(http://cyberhound.com/)


Confidentiality Notice: This email, including any attachments, is confidential and may be privileged.  If you are not the intended recipient please notify the sender immediately and delete it. You should not copy it or use it for any purpose or disclose its contents to any other person without CyberHound's prior written permission. CyberHound Pty Ltd reserves the right to monitor all email communications passing through its networks and devices.

Re: Update to the Pyzor plugin (Re: bug 6108)

[RW <rw...@googlemail.com>]

On Sun, 26 Feb 2017 19:35:11 +1000
James Birkett wrote:

> Hi,
> 
> Bug 6108 says is that the pyzor plugin ignores whitelisting entirely,
> and a comment on that bug suggests updating the Pyzor plugin to use
> Wilson Score formula described here
> http://www.evanmiller.org/how-not-to-sort-by-average-rating.html

SpamAssassin allows easy reporting of whole mailboxes of spam, but to
report ham you have to go to the website and type in captchas (I think
there are a few trusted admins that can bypass this). There's liable to
be a substantial bias in reporting, and the bias is likely to vary
strongly from hash to hash. I don't know for sure, but I suspect that a
lot of whitelisting counts may be historic.

Emails where the body contains only links or addresses get
predigested down to a null string and all have the same hash with a
26.6:1 reporting ratio. There are numerous variants with additional
boilerplate text. To eliminate this class of FP you need a threshold of
at least 0.97.

All in all I don't think it makes sense to use anything other than very
high thresholds. If you want to persue this I would suggest using a
threshold of at least 0.99 and varying the z parameter instead.


> I have tested this using a spreadsheet and our own corpus of spam and
> ham email and got good results,

I think a lot of the benefit you are seeing comes from the fact that
you are rescanning corpus mail rather than using the pyzor counts at
delivery time.

Using Wilson doesn't just affect whitelisted emails it also sets the
minimum number of reports needed for a pyzor hit. For your three rules
that rises from 5 to 381, 22 and 12 (assuming that the very low z value
in the first rule is an error). In production you will lose pyzor hits,
and for new hashes, the most valuable early hits.  

I think pyzor's own defaults are sensible - just ignore anything
that's whitelisted.  


> I wanted to make it possible to have multiple spamassassin rules with
> different wilson score parameters giving different spamassassin
> scores, but obviously we don't want to query pyzor multiple times on
> the same mail so I've changed the actual pyzor lookup to be done
> during extract_metadata() instead
> of when the eval-rule is run. 

The usual way this would be done is to have both functions look for
cached counts and run pyzor if they aren't there - see the BAYES_*
rules for an example.