You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spamassassin.apache.org by Da...@chaosreigns.com on 2011/03/02 03:19:17 UTC
Inconsistent mass-check submissions
I wrote a script that looks at the ruleqa -net reports, and looks for
corpora that were present the previous week but missing the current week
(so if it's missing two weeks in a row, it'll only get printed for the
first missing week). The results do correlate with spikes in the ruleqa
reports, like 20110219.
I believe the reason I show up here is because, due to incompatibility with
the latest LTS Ubuntu release, I'm running mass-check on my home computer,
with a less reliable network connection.
I'm curious what the situations are for other people showing up here,
and if they can be improved.
And what's the status on the mass-check submitters mailing list? Creating
a mailing list should not be so hard. I can set one up on my server if
that would help. It would be ridiculous, but no less so than the amount of
time it has already taken for the list to be created.
20101225
20110101
20110108
20110115
20110122
Missing ham-net-bb-doc.
Missing ham-net-bb-fredt.
Missing ham-net-bb-kmcgrail.
Missing ham-net-bb-trec_enron.
Missing ham-net-bb-zmi.
Missing ham-net-grenier.
Missing spam-net-bb-doc.
Missing spam-net-bb-fredt.
Missing spam-net-bb-kmcgrail.
Missing spam-net-bb-trec_enron.
Missing spam-net-bb-zmi.
Missing spam-net-grenier.
20110129
Missing ham-net-kgolding.
Missing spam-net-kgolding.
20110205
Missing ham-net-llanga.
Missing ham-net-wt-hamtrap-en.
Missing spam-net-llanga.
Missing spam-net-wt-hamtrap-en.
20110212
Missing ham-net-darxus.
Missing ham-net-kgolding.
Missing spam-net-darxus.
Missing spam-net-kgolding.
20110219
Missing ham-net-bernie-it_batt.
Missing ham-net-bernie-mix.
Missing ham-net-dos.
Missing ham-net-jarif.
Missing spam-net-bernie-it_batt.
Missing spam-net-bernie-mix.
Missing spam-net-dos.
Missing spam-net-jarif.
20110226
Missing ham-net-darxus.
Missing spam-net-darxus.
Re: Inconsistent mass-check submissions
Posted by Karsten Bräckelmann <gu...@rudersport.de>.
On Tue, 2011-03-01 at 21:19 -0500, Darxus@chaosreigns.com wrote:
[...]
> And what's the status on the mass-check submitters mailing list? Creating
> a mailing list should not be so hard. [...]
We're depending (and waiting) on the ASF infra team here. I see the
request from end of Jan in my archive, though no response yet. It
probably fell between the cracks somewhere. I'll ping them. Thanks for
the "gentle" (that is after the above pruning) reminder.
--
char *t="\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i<l;i++){ i%8? c<<=1:
(c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}
Re: Inconsistent mass-check submissions
Posted by Jari Fredriksson <ja...@iki.fi>.
On 2.3.2011 4:19, Darxus@chaosreigns.com wrote:
> I wrote a script that looks at the ruleqa -net reports, and looks for
> corpora that were present the previous week but missing the current week
> (so if it's missing two weeks in a row, it'll only get printed for the
> first missing week). The results do correlate with spikes in the ruleqa
> reports, like 20110219.
>
> I believe the reason I show up here is because, due to incompatibility with
> the latest LTS Ubuntu release, I'm running mass-check on my home computer,
> with a less reliable network connection.
>
> I'm curious what the situations are for other people showing up here,
> and if they can be improved.
I lost my server, and it took about 2 weeks to prepare it online from
backups. It's good now.
> Missing ham-net-jarif.
> Missing spam-net-jarif.
>
--
Extreme fear can neither fight nor fly.
-- William Shakespeare, "The Rape of Lucrece"
Re: Inconsistent mass-check submissions
Posted by Dan McDonald <da...@austinenergy.com>.
<Darxus <at> chaosreigns.com> writes:
>
> For this week:
>
> 20110305
> Missing ham-net-danmcdonald.
> Missing ham-net-nbebout.
> Missing spam-net-danmcdonald.
> Missing spam-net-nbebout.
>
I generated the masscheck on 20110305. I think it finally finished around
midnight. Last week I finally got around to sorting out my January and February
ham into my masscheck folders and the nightly runs are taking 12-18 hours
again... i probably got a large ham stuck in there.
--
Dan McDonald, CCIE # 2495, CISSP # 78281
Re: Inconsistent mass-check submissions
Posted by Da...@chaosreigns.com.
For this week:
20110305
Missing ham-net-danmcdonald.
Missing ham-net-nbebout.
Missing spam-net-danmcdonald.
Missing spam-net-nbebout.
The graph of total spam and ham counts in the corpora:
http://www.chaosreigns.com/dnswl/tot.svg
The data file: http://www.chaosreigns.com/dnswl/sa.qa/dnswl.sa.tot.txt
Date Ham Spam
20110226 263567 512001
20110305 188723 358605
Change: -28.4% -30.0%
It's looking like Comcast's way of informing me that the cablemodem with
which they have provided me has been EOLed has been to shut it off for a
while once a week including the time I do -net submissions. I got around
it this time.
--
"The most merciful thing in the world, I think, is the inability of the
human mind to correlate all its contents."
http://www.ChaosReigns.com
Re: Inconsistent mass-check submissions
Posted by John Hardin <jh...@impsec.org>.
On Wed, 2 Mar 2011, Darxus@chaosreigns.com wrote:
> On 03/02, John Hardin wrote:
>> Is the desire for distributed processing stronger than the desire
>> for consistent results? I'd suggest at least part of the problem
>> could be addressed by uploading the spam corpora and letting the
>> central masscheck chew on it. Automating the collect-expire-upload
>> process for corpora is easy and is less sensitive to temporary
>> outages - so what if your uploaded spam corpus is a week stale due
>> to a local failure?
>
> The entire reason for not uploading the spam corpora itself is privacy.
Privacy is why I only mentioned the spam corpora. Can you give me an
example of something in a spam you want kept private?
> However, I'd rather not also need to think about which of my emails might
> contain information too sensitive to upload where someone else might be
> abel to read it. (Even though, really, if it's that sensitive, it
> shouldn't be going over SMTP in cleartext.)
Agreed that the ham corpus has privacy issues making upload to
semi-private storage problematic. I wasn't suggesting that.
> But I have wondered about uploading my spam, and running mass-check on my
> non-spam.
I think the only objections to uploading spam are leaking valid private
email addresses (which the spammers _already have_ or they wouldn't appear
in the spam corpus) and the desire to spread the processing load around.
Are there any objections I'm not seeing?
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
Phobias should not be the basis for laws.
-----------------------------------------------------------------------
11 days until Albert Einstein's 132nd Birthday
Re: Inconsistent mass-check submissions
Posted by Da...@chaosreigns.com.
On 03/02, John Hardin wrote:
> Is the desire for distributed processing stronger than the desire
> for consistent results? I'd suggest at least part of the problem
> could be addressed by uploading the spam corpora and letting the
> central masscheck chew on it. Automating the collect-expire-upload
> process for corpora is easy and is less sensitive to temporary
> outages - so what if your uploaded spam corpus is a week stale due
> to a local failure?
The entire reason for not uploading the spam corpora itself is privacy.
I'm pretty sure everybody agrees that for spamassassin, it would be better
if everybody used the first and easiest method suggested on the nightly
mass-check page, and uploaded their corpora instead of running mass-check
and uploading the results.
http://wiki.apache.org/spamassassin/NightlyMassCheck
However, I'd rather not also need to think about which of my emails might
contain information too sensitive to upload where someone else might be
abel to read it. (Even though, really, if it's that sensitive, it
shouldn't be going over SMTP in cleartext.)
But I have wondered about uploading my spam, and running mass-check on my
non-spam.
--
"Don't go around saying the world owes you a living. The world owes you
nothing. It was here first." - Mark Twain
http://www.ChaosReigns.com
Re: Inconsistent mass-check submissions
Posted by John Hardin <jh...@impsec.org>.
On Tue, 1 Mar 2011, Darxus@chaosreigns.com wrote:
> I believe the reason I show up here is because, due to incompatibility
> with the latest LTS Ubuntu release, I'm running mass-check on my home
> computer, with a less reliable network connection.
>
> I'm curious what the situations are for other people showing up here,
> and if they can be improved.
Is the desire for distributed processing stronger than the desire for
consistent results? I'd suggest at least part of the problem could be
addressed by uploading the spam corpora and letting the central masscheck
chew on it. Automating the collect-expire-upload process for corpora is
easy and is less sensitive to temporary outages - so what if your uploaded
spam corpus is a week stale due to a local failure?
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
The fetters imposed on liberty at home have ever been forged out
of the weapons provided for defense against real, pretended, or
imaginary dangers from abroad. -- James Madison, 1799
-----------------------------------------------------------------------
11 days until Albert Einstein's 132nd Birthday