You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spamassassin.apache.org by Justin Mason <jm...@jmason.org> on 2009/04/01 15:32:59 UTC

Re: nightly mass-checks

link?  these look ok:

http://ruleqa.spamassassin.org/20090331-r760369-n/RCVD_IN_PBL/detail#all

although there are only two submitters at the moment, myself and zmi.
everyone else seems to have given up.

--j.

2009/3/31 Karsten Bräckelmann <gu...@rudersport.de>:
> Guys,
>
> what happened to the nightly mass-checks? With the exception of the
> weekly network mass-checks, they are essentially empty. :-/
>
> Oh, and where's Theo's massive corpus?
>
>
> --
> char *t="\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
> main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i<l;i++){ i%8? c<<=1:
> (c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}
>
>

Re: nightly mass-checks

Posted by Justin Mason <jm...@jmason.org>.
On Tue, Apr 7, 2009 at 09:48, Justin Mason <jm...@jmason.org> wrote:
> On Tue, Apr 7, 2009 at 02:36, Daryl C. W. O'Shea
> <sp...@dostech.ca> wrote:
>> On 06/04/2009 5:35 AM, Justin Mason wrote:
>>> 2009/4/6 Justin Mason <jm...@jmason.org>:
>>>> 2009/4/6 Karsten Bräckelmann <gu...@rudersport.de>:
>>>>> On Sun, 2009-04-05 at 20:59 -0400, Daryl C. W. O'Shea wrote:
>>>>>> Apparently something is broken... I've only checked my mass-check server
>>>>>> output from the last 10 days, but I've certainly uploaded my results for
>>>>>> my million messages each of those days.
>>>>> Possibly related to "Moving heavy I/O-operations"?
>>>> I'll take a look :(
>>>
>>> something's definitely up as a load of them seem to be listed as "zmi" only.
>>> I think it's "forgetting" contributors when newer logs arrive.
>>
>> zmi has in the past submitted logs with the wrong revision number (not
>> one of the nightly revisions)... perhaps he is again and its causing
>> problems.  Although last time it just created an extra line in the
>> interface for his revision.
>>
>
> That may be part of it.  but it's a general bug that needs to be fixed too :(
>
> first off the logs are like this:
>
>    ham-jm @ 20090406-r943854-n
>    ham-dos @ 20090406-r943854-n
>    ham-somebody @ 20090406-r943854-n
>
> the corpus-hourly scripts should run at this point and generate a DETAILS
> hit-frequencies summary using ham-jm, ham-dos and ham-somebody for
> 20090406-r943854-n.
>
> then the day rolls over, some time passes, and jm, dos and somebody uploads
> today's logs:
>
>    ham-jm @ 20090407-r943912-n
>    ham-dos @ 20090407-r943912-n
>    ham-somebody @ 20090407-r943912-n
>
> Now, what should happen is that the previous DETAILS file is still valid.
>
> then zmi uploads his logs for 20090406-r943854-n:
>
>    ham-jm @ 20090407-r943912-n
>    ham-dos @ 20090407-r943912-n
>    ham-somebody @ 20090407-r943912-n
>    ham-zmi @ 20090406-r943854-n
>
> At this point the corpus-hourly scripts run again -- and the only log
> they can see for 20090406-r943854-n is zmi's.  therefore the old info for
> the other 3 submitters gets overwritten!
>
> this is obviously a corpus-hourly bug.  what it should do is keep the old
> logs from the prev run, and be able to add the new zmi log to them (in
> addition).  the fact that it can't is a pretty major omission and I'm
> surprised this hasn't happened before...
>
> anyway, for now I've turned off parsing of zmi's logs to try to help a little.
> but I need to get some tuits to rewrite this logic entirely :(

hooray! got some tuits.  I think this should now be working again, although
it could do with some beautification to remove visible daterev duplication
in the ruleqa output page...

--j.

Re: nightly mass-checks

Posted by Justin Mason <jm...@jmason.org>.
On Tue, Apr 7, 2009 at 02:36, Daryl C. W. O'Shea
<sp...@dostech.ca> wrote:
> On 06/04/2009 5:35 AM, Justin Mason wrote:
>> 2009/4/6 Justin Mason <jm...@jmason.org>:
>>> 2009/4/6 Karsten Bräckelmann <gu...@rudersport.de>:
>>>> On Sun, 2009-04-05 at 20:59 -0400, Daryl C. W. O'Shea wrote:
>>>>> Apparently something is broken... I've only checked my mass-check server
>>>>> output from the last 10 days, but I've certainly uploaded my results for
>>>>> my million messages each of those days.
>>>> Possibly related to "Moving heavy I/O-operations"?
>>> I'll take a look :(
>>
>> something's definitely up as a load of them seem to be listed as "zmi" only.
>> I think it's "forgetting" contributors when newer logs arrive.
>
> zmi has in the past submitted logs with the wrong revision number (not
> one of the nightly revisions)... perhaps he is again and its causing
> problems.  Although last time it just created an extra line in the
> interface for his revision.
>

That may be part of it.  but it's a general bug that needs to be fixed too :(

first off the logs are like this:

    ham-jm @ 20090406-r943854-n
    ham-dos @ 20090406-r943854-n
    ham-somebody @ 20090406-r943854-n

the corpus-hourly scripts should run at this point and generate a DETAILS
hit-frequencies summary using ham-jm, ham-dos and ham-somebody for
20090406-r943854-n.

then the day rolls over, some time passes, and jm, dos and somebody uploads
today's logs:

    ham-jm @ 20090407-r943912-n
    ham-dos @ 20090407-r943912-n
    ham-somebody @ 20090407-r943912-n

Now, what should happen is that the previous DETAILS file is still valid.

then zmi uploads his logs for 20090406-r943854-n:

    ham-jm @ 20090407-r943912-n
    ham-dos @ 20090407-r943912-n
    ham-somebody @ 20090407-r943912-n
    ham-zmi @ 20090406-r943854-n

At this point the corpus-hourly scripts run again -- and the only log
they can see for 20090406-r943854-n is zmi's.  therefore the old info for
the other 3 submitters gets overwritten!

this is obviously a corpus-hourly bug.  what it should do is keep the old
logs from the prev run, and be able to add the new zmi log to them (in
addition).  the fact that it can't is a pretty major omission and I'm
surprised this hasn't happened before...

anyway, for now I've turned off parsing of zmi's logs to try to help a little.
but I need to get some tuits to rewrite this logic entirely :(

--j.

Re: nightly mass-checks

Posted by "Daryl C. W. O'Shea" <sp...@dostech.ca>.
On 06/04/2009 5:35 AM, Justin Mason wrote:
> 2009/4/6 Justin Mason <jm...@jmason.org>:
>> 2009/4/6 Karsten Bräckelmann <gu...@rudersport.de>:
>>> On Sun, 2009-04-05 at 20:59 -0400, Daryl C. W. O'Shea wrote:
>>>> Apparently something is broken... I've only checked my mass-check server
>>>> output from the last 10 days, but I've certainly uploaded my results for
>>>> my million messages each of those days.
>>> Possibly related to "Moving heavy I/O-operations"?
>> I'll take a look :(
> 
> something's definitely up as a load of them seem to be listed as "zmi" only.
> I think it's "forgetting" contributors when newer logs arrive.

zmi has in the past submitted logs with the wrong revision number (not
one of the nightly revisions)... perhaps he is again and its causing
problems.  Although last time it just created an extra line in the
interface for his revision.

Daryl


Re: nightly mass-checks

Posted by Justin Mason <jm...@jmason.org>.
2009/4/6 Justin Mason <jm...@jmason.org>:
> 2009/4/6 Karsten Bräckelmann <gu...@rudersport.de>:
>> On Sun, 2009-04-05 at 20:59 -0400, Daryl C. W. O'Shea wrote:
>>> Apparently something is broken... I've only checked my mass-check server
>>> output from the last 10 days, but I've certainly uploaded my results for
>>> my million messages each of those days.
>>
>> Possibly related to "Moving heavy I/O-operations"?
>
> I'll take a look :(

something's definitely up as a load of them seem to be listed as "zmi" only.
I think it's "forgetting" contributors when newer logs arrive.

--j.

Re: nightly mass-checks

Posted by Justin Mason <jm...@jmason.org>.
2009/4/6 Karsten Bräckelmann <gu...@rudersport.de>:
> On Sun, 2009-04-05 at 20:59 -0400, Daryl C. W. O'Shea wrote:
>> Apparently something is broken... I've only checked my mass-check server
>> output from the last 10 days, but I've certainly uploaded my results for
>> my million messages each of those days.
>
> Possibly related to "Moving heavy I/O-operations"?

I'll take a look :(

Re: nightly mass-checks

Posted by Karsten Bräckelmann <gu...@rudersport.de>.
On Sun, 2009-04-05 at 20:59 -0400, Daryl C. W. O'Shea wrote:
> Apparently something is broken... I've only checked my mass-check server
> output from the last 10 days, but I've certainly uploaded my results for
> my million messages each of those days.

Possibly related to "Moving heavy I/O-operations"?


-- 
char *t="\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i<l;i++){ i%8? c<<=1:
(c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}


Re: nightly mass-checks

Posted by "Daryl C. W. O'Shea" <sp...@dostech.ca>.
On 01/04/2009 9:32 AM, Justin Mason wrote:
> link?  these look ok:
> 
> http://ruleqa.spamassassin.org/20090331-r760369-n/RCVD_IN_PBL/detail#all
> 
> although there are only two submitters at the moment, myself and zmi.
> everyone else seems to have given up.

Apparently something is broken... I've only checked my mass-check server
output from the last 10 days, but I've certainly uploaded my results for
my million messages each of those days.

Daryl


Re: nightly mass-checks

Posted by Karsten Bräckelmann <gu...@rudersport.de>.
On Wed, 2009-04-01 at 14:32 +0100, Justin Mason wrote:
> link?  these look ok:
> 
> http://ruleqa.spamassassin.org/20090331-r760369-n/RCVD_IN_PBL/detail#all
> 
> although there are only two submitters at the moment, myself and zmi.
> everyone else seems to have given up.

Hmm, I phrased that badly. Yes, technically it looks fine. Though what I
saw yesterday was even less than that, only a single corpus wherever I
checked. Like this...
  http://ruleqa.spamassassin.org/20090325-r758178-n

About 1k spam. That I meant by essentially empty. I do recall *mass*
check results 3 orders of magnitude higher.


> 2009/3/31 Karsten Bräckelmann <gu...@rudersport.de>:
> > Guys,
> >
> > what happened to the nightly mass-checks? With the exception of the
> > weekly network mass-checks, they are essentially empty. :-/

FWIW, I actually pondered to contribute to the mass-checks, too,
eventually. Currently blocked by a couple other stuff, including
hardware that can handle it.


-- 
char *t="\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i<l;i++){ i%8? c<<=1:
(c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}


Re: nightly mass-checks

Posted by "Kevin A. McGrail" <km...@pccc.com>.
Not yet dead yet!  Did you get my corpora worked in yet so I can start 
having my rules not fail horribly?

I just synched my latest spam corpora to mailcorpus_kmcgrail.

I'll set this up in cron as soon as it's being used.

Regards,
KAM
----- Original Message ----- 
From: "Justin Mason" <jm...@jmason.org>
To: "Karsten Bräckelmann" <gu...@rudersport.de>
Cc: <de...@spamassassin.apache.org>
Sent: Wednesday, April 01, 2009 9:32 AM
Subject: Re: nightly mass-checks


> link?  these look ok:
>
> http://ruleqa.spamassassin.org/20090331-r760369-n/RCVD_IN_PBL/detail#all
>
> although there are only two submitters at the moment, myself and zmi.
> everyone else seems to have given up.
>
> --j.
>
> 2009/3/31 Karsten Bräckelmann <gu...@rudersport.de>:
>> Guys,
>>
>> what happened to the nightly mass-checks? With the exception of the
>> weekly network mass-checks, they are essentially empty. :-/
>>
>> Oh, and where's Theo's massive corpus?
>>
>>
>> --
>> char 
>> *t="\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
>> main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i<l;i++){ i%8? 
>> c<<=1:
>> (c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ 
>> putchar(t[s]);h=m;s=0; }}}
>>
>>
>