You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spamassassin.apache.org by Jeff Chan <je...@surbl.org> on 2009/08/13 12:46:31 UTC
Re: collecting corpora
On Thursday, July 16, 2009, 1:40:34 PM, Justin Mason wrote:
> One useful factor of ham is that it's not time-sensitive; a mail that
> was ham in 2003 would still be ham today. So we can collect old ham
> mail archives, or submissions of relatively old mail, if necessary.
This may be a false assumption. A spamvertised or spam sending
domain from 2003 could have expired and been re-registered by
a different organization. Same for ham. Both ham and spam
should have expiration times. 1 year would probably be good,
since spamvertised domains probably don't get renewed.
Jeff C.
Re: Time Based Reuse for Rules
Posted by Warren Togami <wt...@redhat.com>.
On 08/13/2009 10:57 AM, Justin Mason wrote:
>>
>> This is a patch to the Reuse plugin I'd like to see, if someone has any
>> ideas please speak up.
>
> it'd work well there. Pretend to be able to reuse previous hits, but
> the actual effect would be to inhibit the rule entirely on old mails.
>
My corpus is completely lacking spamassassin headers.
Warren
Re: Time Based Reuse for Rules
Posted by Justin Mason <jm...@jmason.org>.
On Thu, Aug 13, 2009 at 15:32, Michael Parker<pa...@pobox.com> wrote:
>
> On Aug 13, 2009, at 4:26 AM, Justin Mason wrote:
>
>> On Thu, Aug 13, 2009 at 11:46, Jeff Chan<je...@surbl.org> wrote:
>>>
>>> On Thursday, July 16, 2009, 1:40:34 PM, Justin Mason wrote:
>>>>
>>>> One useful factor of ham is that it's not time-sensitive; a mail that
>>>> was ham in 2003 would still be ham today. So we can collect old ham
>>>> mail archives, or submissions of relatively old mail, if necessary.
>>>
>>> This may be a false assumption. A spamvertised or spam sending
>>> domain from 2003 could have expired and been re-registered by
>>> a different organization. Same for ham. Both ham and spam
>>> should have expiration times. 1 year would probably be good,
>>> since spamvertised domains probably don't get renewed.
>>
>> yep, I was talking with a SURBLer about this last week I think. we
>> should probably add meta conditions ot the URIBL ruleset to ensure
>> they don't fire at all on old messages.
>
> This is a patch to the Reuse plugin I'd like to see, if someone has any
> ideas please speak up.
it'd work well there. Pretend to be able to reuse previous hits, but
the actual effect would be to inhibit the rule entirely on old mails.
--
--j.
Time Based Reuse for Rules
Posted by Michael Parker <pa...@pobox.com>.
On Aug 13, 2009, at 4:26 AM, Justin Mason wrote:
> On Thu, Aug 13, 2009 at 11:46, Jeff Chan<je...@surbl.org> wrote:
>> On Thursday, July 16, 2009, 1:40:34 PM, Justin Mason wrote:
>>> One useful factor of ham is that it's not time-sensitive; a mail
>>> that
>>> was ham in 2003 would still be ham today. So we can collect old ham
>>> mail archives, or submissions of relatively old mail, if necessary.
>>
>> This may be a false assumption. A spamvertised or spam sending
>> domain from 2003 could have expired and been re-registered by
>> a different organization. Same for ham. Both ham and spam
>> should have expiration times. 1 year would probably be good,
>> since spamvertised domains probably don't get renewed.
>
> yep, I was talking with a SURBLer about this last week I think. we
> should probably add meta conditions ot the URIBL ruleset to ensure
> they don't fire at all on old messages.
This is a patch to the Reuse plugin I'd like to see, if someone has
any ideas please speak up.
Michael
>
> --
> --j.
Re: collecting corpora
Posted by Mark Martinec <Ma...@ijs.si>.
> >>>> One useful factor of ham is that it's not time-sensitive; a mail that
> >>>> was ham in 2003 would still be ham today. So we can collect old ham
> >>>> mail archives, or submissions of relatively old mail, if necessary.
> >>>
> >>> This may be a false assumption. A spamvertised or spam sending
> >>> domain from 2003 could have expired and been re-registered by
> >>> a different organization. Same for ham. Both ham and spam
> >>> should have expiration times. 1 year would probably be good,
> >>> since spamvertised domains probably don't get renewed.
> >>
> >> yep, I was talking with a SURBLer about this last week I think. we
> >> should probably add meta conditions ot the URIBL ruleset to ensure
> >> they don't fire at all on old messages.
>
> if we had enough ham to get useful results with that limit, sure. As
> it is, I'm not sure that's the case.
Btw, I just came across this article (from CEAS 2009):
Jose-Marcio Martins da Cruz, Gordon V. Cormack:
Using old Spam and Ham Samples to Train Email Filters
http://www.j-chkmail.org/ceas/ceas09-gvcjm.pdf
Mark
Re: collecting corpora
Posted by Justin Mason <jm...@jmason.org>.
On Fri, Aug 14, 2009 at 01:25, Warren Togami<wt...@redhat.com> wrote:
> On 08/13/2009 07:33 PM, Michael Parker wrote:
>>
>> Historical accuracy of network tests is key, providing corpora without
>> SpamAssassin rules from actual receive time does not help scoring, it
>> hurts it.
>>
>> Michael
>>
>
> Then shouldn't the documentation mention this? My corpora is inconsistently
> filtered by spamassassin in the past. Other corpora from other users
> processed nightly on my server has never been filtered by spamassassin in
> the past.
That statement is mainly Michael's opinion. I'm not sure about it; my
opinion is that we're better off with "post-facto" network test
results from the mass-check, than with hardly any network test results
at all.
The status of your corpora is pretty much the status of most
contributors' corpora, fwiw.
--
--j.
Re: collecting corpora
Posted by Warren Togami <wt...@redhat.com>.
On 08/13/2009 07:33 PM, Michael Parker wrote:
>
> Historical accuracy of network tests is key, providing corpora without
> SpamAssassin rules from actual receive time does not help scoring, it
> hurts it.
>
> Michael
>
Then shouldn't the documentation mention this? My corpora is
inconsistently filtered by spamassassin in the past. Other corpora from
other users processed nightly on my server has never been filtered by
spamassassin in the past.
Warren
Re: collecting corpora
Posted by Michael Parker <pa...@pobox.com>.
On Aug 13, 2009, at 4:07 PM, Warren Togami wrote:
> On 08/13/2009 11:04 AM, Justin Mason wrote:
>>>
>>> IMHO, none of the network tests should be used during masscheck
>>> for ham
>>> older than 4 weeks. Thoughts?
>>>
>>
>> if we had enough ham to get useful results with that limit, sure. As
>> it is, I'm not sure that's the case.
>>
>
> If we are in agreement that old network tests are not good, all we
> are doing is collecting large numbers of bad results and clouding
> the statistical measurements of recent network tests.
>
I wouldn't say that we're in agreement. You're measuring historical
accuracy for reuse results. At that specific point in time domain
example.com was on the SURBL/URIDB/etc list. At actual receive time
how good was the list? Thats what you're measuring. You aren't going
back later and running those domains through the current lists.
Historical accuracy of network tests is key, providing corpora without
SpamAssassin rules from actual receive time does not help scoring, it
hurts it.
Michael
Re: collecting corpora
Posted by Warren Togami <wt...@redhat.com>.
On 08/13/2009 11:04 AM, Justin Mason wrote:
>>
>> IMHO, none of the network tests should be used during masscheck for ham
>> older than 4 weeks. Thoughts?
>>
>
> if we had enough ham to get useful results with that limit, sure. As
> it is, I'm not sure that's the case.
>
If we are in agreement that old network tests are not good, all we are
doing is collecting large numbers of bad results and clouding the
statistical measurements of recent network tests.
Warren
Re: collecting corpora
Posted by Warren Togami <wt...@redhat.com>.
On 08/13/2009 11:04 AM, Justin Mason wrote:
>>>
>>> yep, I was talking with a SURBLer about this last week I think. we
>>> should probably add meta conditions ot the URIBL ruleset to ensure
>>> they don't fire at all on old messages.
>>>
>>
>> IMHO, none of the network tests should be used during masscheck for ham
>> older than 4 weeks. Thoughts?
>>
>
> if we had enough ham to get useful results with that limit, sure. As
> it is, I'm not sure that's the case.
>
Make it 3 months then?
Point being, URIBL's and DNSBL's both become increasingly less accurate
over time. And there is no reason to expect the network hash servers to
know hashes from a year ago. Things might be worse with all network tests?
Warren
Re: collecting corpora
Posted by Justin Mason <jm...@jmason.org>.
On Thu, Aug 13, 2009 at 16:00, Warren Togami<wt...@redhat.com> wrote:
> On 08/13/2009 07:26 AM, Justin Mason wrote:
>>
>> On Thu, Aug 13, 2009 at 11:46, Jeff Chan<je...@surbl.org> wrote:
>>>
>>> On Thursday, July 16, 2009, 1:40:34 PM, Justin Mason wrote:
>>>>
>>>> One useful factor of ham is that it's not time-sensitive; a mail that
>>>> was ham in 2003 would still be ham today. So we can collect old ham
>>>> mail archives, or submissions of relatively old mail, if necessary.
>>>
>>> This may be a false assumption. A spamvertised or spam sending
>>> domain from 2003 could have expired and been re-registered by
>>> a different organization. Same for ham. Both ham and spam
>>> should have expiration times. 1 year would probably be good,
>>> since spamvertised domains probably don't get renewed.
>>
>> yep, I was talking with a SURBLer about this last week I think. we
>> should probably add meta conditions ot the URIBL ruleset to ensure
>> they don't fire at all on old messages.
>>
>
> IMHO, none of the network tests should be used during masscheck for ham
> older than 4 weeks. Thoughts?
>
if we had enough ham to get useful results with that limit, sure. As
it is, I'm not sure that's the case.
--
--j.
Re: collecting corpora
Posted by Warren Togami <wt...@redhat.com>.
On 08/13/2009 07:26 AM, Justin Mason wrote:
> On Thu, Aug 13, 2009 at 11:46, Jeff Chan<je...@surbl.org> wrote:
>> On Thursday, July 16, 2009, 1:40:34 PM, Justin Mason wrote:
>>> One useful factor of ham is that it's not time-sensitive; a mail that
>>> was ham in 2003 would still be ham today. So we can collect old ham
>>> mail archives, or submissions of relatively old mail, if necessary.
>>
>> This may be a false assumption. A spamvertised or spam sending
>> domain from 2003 could have expired and been re-registered by
>> a different organization. Same for ham. Both ham and spam
>> should have expiration times. 1 year would probably be good,
>> since spamvertised domains probably don't get renewed.
>
> yep, I was talking with a SURBLer about this last week I think. we
> should probably add meta conditions ot the URIBL ruleset to ensure
> they don't fire at all on old messages.
>
IMHO, none of the network tests should be used during masscheck for ham
older than 4 weeks. Thoughts?
Warren Togami
wtogami@redhat.com
Re: collecting corpora
Posted by Justin Mason <jm...@jmason.org>.
On Thu, Aug 13, 2009 at 11:46, Jeff Chan<je...@surbl.org> wrote:
> On Thursday, July 16, 2009, 1:40:34 PM, Justin Mason wrote:
>> One useful factor of ham is that it's not time-sensitive; a mail that
>> was ham in 2003 would still be ham today. So we can collect old ham
>> mail archives, or submissions of relatively old mail, if necessary.
>
> This may be a false assumption. A spamvertised or spam sending
> domain from 2003 could have expired and been re-registered by
> a different organization. Same for ham. Both ham and spam
> should have expiration times. 1 year would probably be good,
> since spamvertised domains probably don't get renewed.
yep, I was talking with a SURBLer about this last week I think. we
should probably add meta conditions ot the URIBL ruleset to ensure
they don't fire at all on old messages.
--
--j.