You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spamassassin.apache.org by Justin Mason <jm...@jmason.org> on 2005/11/22 05:38:05 UTC
Re: 3.0.5 rescoring
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Sidney Markowitz writes:
> On 11/22/05 5:14 PM, Loren Wilton wrote:
> >I believe the claim was that it produced
> > reasonable scoring with less effort than the normal method
>
> The less effort is in the procedure to gather the scoring copora from a
> group of users. It's too late to consider that now. Warren's biggest
> problem is that he doesn't have a suitable corpus to rerun the scoring
> on. If he did, the perceptron scripts would work fine for his purpose.
>
> What he really needs, I think, is some people with good corpora willing
> to run a mass check using the 3.0.5 branch.
well, it's more than that. with a small number of corpora, the
scores will be over-optimised for those people. It's a tricky
problem....
- --j.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Exmh CVS
iD8DBQFDgqCtMJF5cimLx9ARAiPVAJ4+sJZhicAPTK8sAowitHrDHx6kpgCfdgSr
5DmbDQJEJvAKyXZlFHavtmM=
=n3FJ
-----END PGP SIGNATURE-----
Re: 3.0.5 rescoring
Posted by Doc Schneider <ma...@maddoc.net>.
Theo Van Dinter wrote:
> On Mon, Nov 21, 2005 at 08:38:05PM -0800, Justin Mason wrote:
>
>>well, it's more than that. with a small number of corpora, the
>>scores will be over-optimised for those people. It's a tricky
>>problem....
>
>
> I've actually been thinking about this a bit. Our normal mass-check runs
> are heavily weighted towards a small number of people already. For 3.1,
> we used 9 people's logs. It totalled 1766844 messages (bmenschel's
> wasn't included apparently). Breaking it down:
>
> Percent Provider
> ------- ----------
> 33.93 jm
> 31.00 theo
> 9.35 daf
> 7.68 rod
> 6.05 parkerm
> 5.62 bzoetekouw
> 5.11 quinlan
> 1.20 cthielen
> 0.07 misak
>
> So basically Justin is 34%, I'm 31%, and everyone else combined is 35%.
> So in reality, the scores are far more tuned for Justin and myself than
> any other single person.
>
> This is something I've been trying to think about wrt doing weekly score
> generations for use by sa-update, but no real solution has come to mind yet.
>
Odd I'm not in there. I should be as doc (for my rsync corpus) maybe I
need to send all my corpus again.
-Doc
Re: 3.0.5 rescoring
Posted by Warren Togami <wt...@redhat.com>.
Theo Van Dinter wrote:
> On Tue, Nov 22, 2005 at 10:17:35PM -0500, Warren Togami wrote:
>
>>I personally want 3.0.5 as a final maintenance release of the 3.0.x
>>series, and for development schedule reasons I need it by Nov 29th at
>>the latest.
>
>
> IMO, since 3.0 will need 3 different mass-check runs, with the last two
> relying on the first one, there's pretty much no way to get it done by
> the 29th.
>
There is a high probability that I will be able to push an update with a
new scoreset after this month. This soon deadline is only for a beta.
I am asking for approval now.
(And yes I do agree sa-update is the ideal way to handle this in the
future as we can better dynamically adapt to rapidly changing spam
threats. I will focus my efforts toward that after we have 3.0.5
finalized and I no longer need to maintain the old stuff anymore.)
Warren Togami
wtogami@redhat.com
Re: 3.0.5 rescoring
Posted by Warren Togami <wt...@redhat.com>.
Theo Van Dinter wrote:
> On Tue, Nov 22, 2005 at 10:17:35PM -0500, Warren Togami wrote:
>
>>I personally want 3.0.5 as a final maintenance release of the 3.0.x
>>series, and for development schedule reasons I need it by Nov 29th at
>>the latest.
>
>
> IMO, since 3.0 will need 3 different mass-check runs, with the last two
> relying on the first one, there's pretty much no way to get it done by
> the 29th.
>
I have approval to use the updated scores when it becomes available.
Can we go ahead with a full mass check for 3.0.5? 3.0.5 can be the
final maintenance release of the 3.0.x series with a fresh set of scores.
I assume this would need someone to drive the process?
Warren Togami
wtogami@redhat.com
Re: 3.0.5 rescoring
Posted by Theo Van Dinter <fe...@apache.org>.
On Tue, Nov 22, 2005 at 10:17:35PM -0500, Warren Togami wrote:
> I personally want 3.0.5 as a final maintenance release of the 3.0.x
> series, and for development schedule reasons I need it by Nov 29th at
> the latest.
IMO, since 3.0 will need 3 different mass-check runs, with the last two
relying on the first one, there's pretty much no way to get it done by
the 29th.
--
Randomly Generated Tagline:
"I could nail your head to the table, set fire to it, and feed the charred
remains to the pak-mara. But we can't always get what we want."
- Sheridan on Babylon 5
Re: 3.0.5 rescoring
Posted by Warren Togami <wt...@redhat.com>.
Doc Schneider wrote:
> Theo Van Dinter wrote:
>
>> On Tue, Nov 22, 2005 at 04:01:19PM -0800, Justin Mason wrote:
>>
>>> by the way, I've decided I'll run a 3.0.5 mass-check on my corpora, if
>>> needed.
>>
>>
>>
>> I could probably do a small one as well if we want to set up something
>> official. The main issue, iirc, is that 3.0 required the full 3 sets
>> of mass-check runs since reuse went in for 3.1. So the time to do it
>> is going to be around 1-2 weeks.
>>
>> I'm tempted to say not to do it for a 3.0.5 release, but spend time
>> figuring
>> out how to get more frequent runs done, and start putting out new
>> scores via
>> sa-update (while not included with the 3.0 releases, sa-update ought
>> to work
>> with 2.x and above).
>>
>
> I'm wondering if you all did scoring test for 2.63 then a new one for 2.64?
>
> I'd think if you didn't (since 2.64 was basically a bug fix) than is a
> new one needed for 3.0.5? Just my thoughts.
>
> Even though it would probably not be totally required for 3.0.5, I can
> also help with a masscheck if you all really think one is needed?
>
> -Doc
There is no Right answer to this. I personally think it would be nice
if we could do something to improve the scores of 3.0.5 because they are
more than a year old.
I personally want 3.0.5 as a final maintenance release of the 3.0.x
series, and for development schedule reasons I need it by Nov 29th at
the latest.
Warren Togami
wtogami@redhat.com
Re: 3.0.5 rescoring
Posted by Doc Schneider <ma...@maddoc.net>.
Theo Van Dinter wrote:
> On Tue, Nov 22, 2005 at 04:01:19PM -0800, Justin Mason wrote:
>
>>by the way, I've decided I'll run a 3.0.5 mass-check on my corpora, if
>>needed.
>
>
> I could probably do a small one as well if we want to set up something
> official. The main issue, iirc, is that 3.0 required the full 3 sets
> of mass-check runs since reuse went in for 3.1. So the time to do it
> is going to be around 1-2 weeks.
>
> I'm tempted to say not to do it for a 3.0.5 release, but spend time figuring
> out how to get more frequent runs done, and start putting out new scores via
> sa-update (while not included with the 3.0 releases, sa-update ought to work
> with 2.x and above).
>
I'm wondering if you all did scoring test for 2.63 then a new one for 2.64?
I'd think if you didn't (since 2.64 was basically a bug fix) than is a
new one needed for 3.0.5? Just my thoughts.
Even though it would probably not be totally required for 3.0.5, I can
also help with a masscheck if you all really think one is needed?
-Doc
Re: 3.0.5 rescoring
Posted by Theo Van Dinter <fe...@apache.org>.
On Tue, Nov 22, 2005 at 04:01:19PM -0800, Justin Mason wrote:
> by the way, I've decided I'll run a 3.0.5 mass-check on my corpora, if
> needed.
I could probably do a small one as well if we want to set up something
official. The main issue, iirc, is that 3.0 required the full 3 sets
of mass-check runs since reuse went in for 3.1. So the time to do it
is going to be around 1-2 weeks.
I'm tempted to say not to do it for a 3.0.5 release, but spend time figuring
out how to get more frequent runs done, and start putting out new scores via
sa-update (while not included with the 3.0 releases, sa-update ought to work
with 2.x and above).
--
Randomly Generated Tagline:
bpi: Gambling term, as in, "You bet your bpi."
Re: 3.0.5 rescoring
Posted by Warren Togami <wt...@redhat.com>.
Theo Van Dinter wrote:
> On Mon, Nov 21, 2005 at 08:38:05PM -0800, Justin Mason wrote:
>
>>well, it's more than that. with a small number of corpora, the
>>scores will be over-optimised for those people. It's a tricky
>>problem....
>
>
> I've actually been thinking about this a bit. Our normal mass-check runs
> are heavily weighted towards a small number of people already. For 3.1,
> we used 9 people's logs. It totalled 1766844 messages (bmenschel's
> wasn't included apparently). Breaking it down:
>
> Percent Provider
> ------- ----------
> 33.93 jm
> 31.00 theo
> 9.35 daf
> 7.68 rod
> 6.05 parkerm
> 5.62 bzoetekouw
> 5.11 quinlan
> 1.20 cthielen
> 0.07 misak
>
> So basically Justin is 34%, I'm 31%, and everyone else combined is 35%.
> So in reality, the scores are far more tuned for Justin and myself than
> any other single person.
>
> This is something I've been trying to think about wrt doing weekly score
> generations for use by sa-update, but no real solution has come to mind yet.
>
We seriously need to improve documentation and tools to make it easier
for people to understand and do this. At our company we need to almost
cripple our Asian office spamassassin because of the FP levels. We need
better representation especially from non-Western users in mass checks.
I for example am trying to get a few native Japanese employees at my
office to participate because of the total lack of Asian representation
currently in mass check. They misunderstood the sorting directions at
first, so I need to train them myself to make sure they do a good job at it.
Warren Togami
wtogami@redhat.com
Re: 3.0.5 rescoring
Posted by Michael Monnerie <m....@zmi.at>.
On Mittwoch, 30. November 2005 19:53 Warren Togami wrote:
> Documenting calling a script is the easy part. Making sure that all
> reporters are following the rules of spam sorting, that is not easy.
What about an IMAP server, where each reporter gets an account on, and
can drop her SPAM to? Like this, you have that reported SPAM locally,
and can doublecheck it. For the reporter, that's a very easy way of
reporting.
I use such a thing for me personally. SPAM received is dropped to a
folder, and after manual checking moved to a SPAM_learn box
(automatically learned and deleted) and a SPAM_sure box, there being
kept forever.
mfg zmi
--
// Michael Monnerie, Ing.BSc --- it-management Michael Monnerie
// http://zmi.at Tel: 0660/4156531 Linux 2.6.11
// PGP Key: "lynx -source http://zmi.at/zmi2.asc | gpg --import"
// Fingerprint: EB93 ED8A 1DCD BB6C F952 F7F4 3911 B933 7054 5879
// Keyserver: www.keyserver.net Key-ID: 0x70545879
Re: 3.0.5 rescoring
Posted by Warren Togami <wt...@redhat.com>.
Michael Monnerie wrote:
> On Dienstag, 22. November 2005 06:44 Theo Van Dinter wrote:
>
>>So basically Justin is 34%, I'm 31%, and everyone else combined is
>>35%.
>
>
> I could send you my hand sorted SPAM, if you like. It's only ~3000
> SPAMs, but maybe worth it - more and more german language SPAM coming
> into my honeypots.
>
> For a more differentiated SPAM score, more people would have to commit
> their SPAM. If that process would be well documented, and people
> encouraged to do so, and the process is as easy as calling a script, I
> believe you could have a lot of reporters.
>
> mfg zmi
Documenting calling a script is the easy part. Making sure that all
reporters are following the rules of spam sorting, that is not easy.
Warren
Re: 3.0.5 rescoring
Posted by Michael Monnerie <m....@zmi.at>.
On Dienstag, 22. November 2005 06:44 Theo Van Dinter wrote:
> So basically Justin is 34%, I'm 31%, and everyone else combined is
> 35%.
I could send you my hand sorted SPAM, if you like. It's only ~3000
SPAMs, but maybe worth it - more and more german language SPAM coming
into my honeypots.
For a more differentiated SPAM score, more people would have to commit
their SPAM. If that process would be well documented, and people
encouraged to do so, and the process is as easy as calling a script, I
believe you could have a lot of reporters.
mfg zmi
--
// Michael Monnerie, Ing.BSc --- it-management Michael Monnerie
// http://zmi.at Tel: 0660/4156531 Linux 2.6.11
// PGP Key: "lynx -source http://zmi.at/zmi2.asc | gpg --import"
// Fingerprint: EB93 ED8A 1DCD BB6C F952 F7F4 3911 B933 7054 5879
// Keyserver: www.keyserver.net Key-ID: 0x70545879
Re: 3.0.5 rescoring
Posted by Theo Van Dinter <fe...@apache.org>.
On Mon, Nov 21, 2005 at 08:38:05PM -0800, Justin Mason wrote:
> well, it's more than that. with a small number of corpora, the
> scores will be over-optimised for those people. It's a tricky
> problem....
I've actually been thinking about this a bit. Our normal mass-check runs
are heavily weighted towards a small number of people already. For 3.1,
we used 9 people's logs. It totalled 1766844 messages (bmenschel's
wasn't included apparently). Breaking it down:
Percent Provider
------- ----------
33.93 jm
31.00 theo
9.35 daf
7.68 rod
6.05 parkerm
5.62 bzoetekouw
5.11 quinlan
1.20 cthielen
0.07 misak
So basically Justin is 34%, I'm 31%, and everyone else combined is 35%.
So in reality, the scores are far more tuned for Justin and myself than
any other single person.
This is something I've been trying to think about wrt doing weekly score
generations for use by sa-update, but no real solution has come to mind yet.
--
Randomly Generated Tagline:
"We don't make mistakes ... We just have happy little accidents."
- The Joy of Bikini Waxing (HardCore TV)