You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Damian <sp...@arcsin.de> on 2018/12/22 23:39:02 UTC
Is the SA Bayes implementation mathematically sound?
Hi all,
is there someone who has a good grasp around the mathematics of Bayes
learning with respect to SpamAssassin?
I assume that training a fresh BayesStore with a set of spam and ham
samples is mathematically sound. What bothers me a little is the
expiration logic.
The purpose of expiration seems to be a practical one, we don't want the
BayesStore grow too much. But is there a conceptual counterpart? One
such concept could be:
Maintain the store as if it were trained from scratch with spam and ham
mails up to N days into the past.
However if I am not mistaken, that is not the implementation.
The nspam and nham magic counters mostly only increase. They will
decrease when a message is forgotten or relearnt, but they will not
decrease on expiration.
If I am not mistaken there are conceptual differences between some
BayesStore implementations. PgSQL will expire tokens if configured, but
it will not expire seen messages. Redis on the other hand expires both
tokens and seen messages (with a huge ttl difference between those two
in the default configuration, on top of that).
As a result, after some time, probably most BayesStores are in a state
for which there is no mail-sample set that can lead to said state via
training. Can such state still lead to statistically valid conclusions?
Can both implementations be correct?
Damian
Re: Is the SA Bayes implementation mathematically sound?
Posted by Rick Macdougall <ri...@ummm-beer.com>.
On 2018-12-24 12:39 p.m., Ian Zimmerman wrote:
> On 2018-12-23 17:02, Rick Macdougall wrote:
>
>> I'm just going to jump in here and mention that I train my bayes in SA
>> and in Thunderbird email client.
>>
>> Thunderbird catches 99%+ and SA catches under 60% with the same
>> training data.
>
> Have you also compared the rates of False Positives?
>
I've seen one FP in Thunderbird in the last 5 years.
I haven't checked Bayes only FP in SA, but I will.
Regards,
Rick
Re: Is the SA Bayes implementation mathematically sound?
Posted by Ian Zimmerman <it...@very.loosely.org>.
On 2018-12-23 17:02, Rick Macdougall wrote:
> I'm just going to jump in here and mention that I train my bayes in SA
> and in Thunderbird email client.
>
> Thunderbird catches 99%+ and SA catches under 60% with the same
> training data.
Have you also compared the rates of False Positives?
--
Please don't Cc: me privately on mailing lists and Usenet,
if you also post the followup to the list or newsgroup.
To reply privately _only_ on Usenet and on broken lists
which rewrite From, fetch the TXT record for no-use.mooo.com.
Re: Is the SA Bayes implementation mathematically sound?
Posted by Rick Macdougall <ri...@ummm-beer.com>.
Hi,
I'm just going to jump in here and mention that I train my bayes in SA
and in Thunderbird email client.
Thunderbird catches 99%+ and SA catches under 60% with the same training
data.
Take that as you will because, of course, there is mail that Thunderbird
never sees due to SA rejecting emails.
Regards,
Rick
Re: Is the SA Bayes implementation mathematically sound?
Posted by Bill Cole <sa...@billmail.scconsult.com>.
On 22 Dec 2018, at 18:39, Damian wrote:
> Hi all,
>
> is there someone who has a good grasp around the mathematics of Bayes
> learning with respect to SpamAssassin?
Justin Mason would be the best person to discuss this. I do not know if
he still reads this list.
> I assume that training a fresh BayesStore with a set of spam and ham
> samples is mathematically sound.
Nope.
I mean, it probably is sound for the initial static set of spam and ham
it is trained with, until more training and expiration happens. So what?
It will NEVER be a mathematically sound Bayesian classifier for the mail
it is asked to classify. Never.
It is imperfect for any ongoing collection of spam and ham. There is no
such thing as a valid sample of email which applies to the spam/ham
classification of tomorrow's email. There are significant qualitative
and quantitative differences over time for any target and across targets
in any period of time. Exactly identical messages sent to multiple
addresses may be ham for one target and spam for another.
> What bothers me a little is the
> expiration logic.
Again the question is: so what?
As is shown almost every week on this list and almost every morning in
the update to the default rules channel, spam is a moving target. As
investment managers are required to say in the US: past performance is
not an indicator of future results.
The "Bayes" classifier SA is an empirically useful tool, not an academic
project. A better implementation might be one that conforms more
rigorously to the underlying math, or it might not. A better
implementation would do a better job classifying today's mail based on
whatever training it has and remembers than the existing implementation.
Re: Is the SA Bayes implementation mathematically sound?
Posted by Damian <sp...@arcsin.de>.
Am 23.12.18 um 02:35 schrieb RW:
>> The purpose of expiration seems to be a practical one, we don't want
>> the BayesStore grow too much. But is there a conceptual counterpart?
>> One such concept could be:
>> Maintain the store as if it were trained from scratch with spam and
>> ham mails up to N days into the past.
>
> If you want to do that you can keep spam and ham corpora, trim them
> periodically and recreate the database from scratch. This works for
> manually maintained corpora, but wont scale to high volume
> auto-training.
Yes, scalability is an issue, but mostly a practical one. My question is
rather about the mathematical foundations of the classifier and whether
SpamAssassin has lost track of its roots. If there were a statistics
exercise with a very tiny corpus and very short mails and the professor
wanted us to calculate the probability of a tiny mail being ham or spam
via Bayes theorem, then there would be only one correct solution.
If the professor wanted to reuse the exercise with different parameters
for various exams, maybe he would tell his assistant to create a little
software so that he could play around with training corpora and tiny
input mails in order to make the solution look elegant for an exam. This
software needs to be correct.
I just presume that SpamAssassin, at its core, along with the naive
Bayes combiner, might be such a software, as long as there is no
expiration involved and while ignoring usual precision/rounding errors.
Assume we take the professor's elegant exercise, feed it to a
stripped-down version of SpamAssassin that outputs probabilities instead
of scores, and see that the probability is mathematically correct.
Now we throw away all tokens and seen messages, and relearn one spam and
ham each. We query the software for a solution for the same input mail
as before. Will it coincide with the correct one? Probably not. We can
make a game out of it. Prepare carefully designed corpora and test input
mails and see how far off the solution can get. I believe that it can
get pretty far off with small corpora, and so did the SpamAssassin
authors when they introduced min_spam_num and min_ham_num. The more
important questions are:
Is there a useful guaranteed boundary of how far off one can get?
There is a boundary of +-100%, but that is not useful.
Is the boundary decreasing with respect to nham+nspam counts, or
is the boundary decreasing with respect to the number of tokens?
> I do it on my own mail for a different reason, I like the idea of being
> able to remove the influence of very old email.
>
>> However if I am not mistaken, that is not the implementation.
>>
>> The nspam and nham magic counters mostly only increase. They will
>> decrease when a message is forgotten or relearnt, but they will not
>> decrease on expiration.
>
> Nor should they as that would affect the frequencies of the tokens
> that haven't been expired. Core tokens that are never expired produce
> the same frequencies they would have had if no expiry had taken place.
> That's the big advantage of token expiry.
This seems plausible to me, but one plausible feature does not prove the
correctness of the whole system in the above sense. I have another,
hopefully plausible design argument that is not implemented:
There might be tokens that are very rare, but when they occur, the mail
is almost certainly spam. Because they are so rare, they always expire
before they can help classify other mails. So shouldn't SpamAssassin
also keep the strongest tokens?
> Ideally expiry should be light enough to only remove ephemeral tokens
> that wont be seen in future and very rare tokens that hardly every
> change anything. Beyond that it's just a compromise. Those that do have
> to compromise on retention aren't going to want a huge reduction in
> return for a minor change in theoretical correctness.
Does this mean that there has been evaluation how minor or non-minor
such change can theoretically be?
>> If I am not mistaken there are conceptual differences between some
>> BayesStore implementations. PgSQL will expire tokens if configured,
>> but it will not expire seen messages. Redis on the other hand expires
>> both tokens and seen messages (with a huge ttl difference between
>> those two in the default configuration, on top of that).
>
> The seen message information is just a flag that keeps track of
> whether a particular email was previous trained as spam or ham;
> expiring it prevents very old emails from being forgotten or
> retrained correctly. It doesn't have anything to do with classification.
After a seen messageid has expired, it can be relearnt, and when it is
relearnt, it increases nspam or nham and its token counters. So it does
have to do with classification in an indirect sense.
Re: Is the SA Bayes implementation mathematically sound?
Posted by RW <rw...@googlemail.com>.
On Sun, 23 Dec 2018 00:39:02 +0100
Damian wrote:
> Hi all,
>
> is there someone who has a good grasp around the mathematics of Bayes
> learning with respect to SpamAssassin?
>
> I assume that training a fresh BayesStore with a set of spam and ham
> samples is mathematically sound.
It's not so sound that a lot of guesswork, and trial and error weren't
needed.
> What bothers me a little is the
> expiration logic.
>
> The purpose of expiration seems to be a practical one, we don't want
> the BayesStore grow too much. But is there a conceptual counterpart?
> One such concept could be:
> Maintain the store as if it were trained from scratch with spam and
> ham mails up to N days into the past.
If you want to do that you can keep spam and ham corpora, trim them
periodically and recreate the database from scratch. This works for
manually maintained corpora, but wont scale to high volume
auto-training.
I do it on my own mail for a different reason, I like the idea of being
able to remove the influence of very old email.
> However if I am not mistaken, that is not the implementation.
>
> The nspam and nham magic counters mostly only increase. They will
> decrease when a message is forgotten or relearnt, but they will not
> decrease on expiration.
Nor should they as that would affect the frequencies of the tokens
that haven't been expired. Core tokens that are never expired produce
the same frequencies they would have had if no expiry had taken place.
That's the big advantage of token expiry.
Ideally expiry should be light enough to only remove ephemeral tokens
that wont be seen in future and very rare tokens that hardly every
change anything. Beyond that it's just a compromise. Those that do have
to compromise on retention aren't going to want a huge reduction in
return for a minor change in theoretical correctness.
> If I am not mistaken there are conceptual differences between some
> BayesStore implementations. PgSQL will expire tokens if configured,
> but it will not expire seen messages. Redis on the other hand expires
> both tokens and seen messages (with a huge ttl difference between
> those two in the default configuration, on top of that).
The seen message information is just a flag that keeps track of
whether a particular email was previous trained as spam or ham;
expiring it prevents very old emails from being forgotten or
retrained correctly. It doesn't have anything to do with classification.