You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spamassassin.apache.org by Doc Schneider <ma...@maddoc.net> on 2006/03/01 05:29:10 UTC
Nightly mass-checks
Gang,
I'm working on a better wiki page for nightly mass-checks and an easier
way for those old and new to SA to be able to do these.
A few things which I'm not entirely sure of are:
1) Command line to run a mass-check $./mass-check --all --progress
ham:mbox:path/to/ham (does this need a / or a /* to run through a whole
directory of mbox ham files? Ditto with spam?)
2) Command line for maildir files? ham:dir:/path/to/ham (Wondering the
same on this also about needing a / or /*)
3) A good example of how to do a mass-check using the SVN method. (Theo
gave me some clues on using it so am pretty much set for doing this.)
4) These is no 4
5) Anyone having a particular problem that should be addressed for doing
mass-checks?
Will post more as I run into different issues.
Thanks!
--
-Doc
SA/SARE -- Ninja
10:15pm up 38 days, 19:36, 15 users, load average: 0.55, 0.48, 0.48
SARE HQ http://www.rulesemporium.com/
Re: Nightly mass-checks
Posted by Bas Zoetekouw <ba...@zoetekouw.net>.
Hi Michael!
You wrote:
> On Donnerstag, 2. März 2006 07:05 Duncan Findlay wrote:
> > - Use -j2 since you have 2 processors... might as well use them.
>
> Wouldn't it be even faster with -j3 or -j4? I know it doesn't help for
> the CPU calculating stuff, but it helps satisfying the CPU when one
> process waits for network I/O, during which it can then run another
> calculating thingy. When running only local tests, -j2 is of course the
> best choice. But that should be *very* much depending on the setup, of
> course, but it may be worth a try.
Yes, for net tests -j can be much higher, if you have the memory to
spare. I use -j12 for the weekly net runs, on a single processor, which
really helps a lot.
--
Kind regards,
+----------------------------------------------------------------------+
| Bas Zoetekouw | Sweet day, so cool, so calm, so bright, |
|----------------------------| The bridall of the earth and skie: |
| bas@o2w.nl, bas@debian.org | The dew shall weep thy fall tonight; |
+----------------------------| For thou must die. |
+-----------------------------------------+
Re: Nightly mass-checks
Posted by Michael Monnerie <m....@zmi.at>.
On Donnerstag, 2. März 2006 07:05 Duncan Findlay wrote:
> - Use -j2 since you have 2 processors... might as well use them.
Wouldn't it be even faster with -j3 or -j4? I know it doesn't help for
the CPU calculating stuff, but it helps satisfying the CPU when one
process waits for network I/O, during which it can then run another
calculating thingy. When running only local tests, -j2 is of course the
best choice. But that should be *very* much depending on the setup, of
course, but it may be worth a try.
mfg zmi
--
// Michael Monnerie, Ing.BSc --- it-management Michael Monnerie
// http://zmi.at Tel: 0660/4156531 Linux 2.6.11
// PGP Key: "lynx -source http://zmi.at/zmi2.asc | gpg --import"
// Fingerprint: EB93 ED8A 1DCD BB6C F952 F7F4 3911 B933 7054 5879
// Keyserver: www.keyserver.net Key-ID: 0x70545879
Re: Nightly mass-checks
Posted by Theo Van Dinter <fe...@apache.org>.
On Thu, Mar 02, 2006 at 12:07:33AM -0600, Doc Schneider wrote:
> I'll try the -j 2 and what does -n do? Already using --progress.
-n tells mass-check not to try sorting the messages by time and also not
to interleave ham and spam. it will likely not make the run faster,
but it'll make starting the run a little faster. (You should not use
-n when running with bayes, but otherwise the ordering doesn't matter.)
--
Randomly Generated Tagline:
A poor excuse is better than no excuse! <grin>
Re: Nightly mass-checks
Posted by Theo Van Dinter <fe...@apache.org>.
On Thu, Mar 02, 2006 at 12:07:33AM -0600, Doc Schneider wrote:
> >>>>is what I'm personally using. The machine is a dual 500 with a gig of
> >>>>RAM. And perl 5.8.6 on it. Anyone have any ideas?
> >>Total size of the files? or how many messages in each mbox?
> >>
> >>A bit over 100k messages total
> >>spams:63731 hams:39385 give or take.
Just for comparison, my current nightly runs, which happen on a dual 2GHz
Opteron, are running at about 8 msgs/sec/processor, which is a bit slower than
it used to be.
For ease of calculation, let's assume msgs/sec is linear based on CPU speed.
Therefore your 500MHz system will be able to do 2 msgs/sec/processor.
100000 messages / 4 msgs/sec = 25000 seconds / 3600 sec/hr = 6.94 hours.
--
Randomly Generated Tagline:
"I know it's weird, but it does make it easier to write poetry in perl. :-)"
- Larry Wall
Re: Nightly mass-checks
Posted by Doc Schneider <ma...@maddoc.net>.
Theo Van Dinter wrote:
> On Wed, Mar 01, 2006 at 11:50:58PM -0600, Doc Schneider wrote:
>>>> ./mass-check --progress --all \
>>>> ham:mbox:/home/masschecker/mail/ham \
>>>> spam:mbox:/home/masschecker/mail/spam
>>>>
>>>> is what I'm personally using. The machine is a dual 500 with a gig of
>>>> RAM. And perl 5.8.6 on it. Anyone have any ideas?
>> Total size of the files? or how many messages in each mbox?
>>
>> A bit over 100k messages total
>> spams:63731 hams:39385 give or take.
>
> Hrm. The first thing that comes to mind is run with "-n -j 2". The -n part
> won't speed up a lot, the -j 2 will tell mass-check to use both processors.
>
> I'd also run with --progress so you can see how far it's gotten. 500MHz isn't
> super fast unfortunately. You may have to trim down the number of messages
> you're planning to process.
>
I do have a 1.8 Ghz box with 1.5 Gigs of RAM but it is in use for other
things.
I'll try the -j 2 and what does -n do? Already using --progress.
Going to grab a new copy of the nightly mass-check and start over.
--
-Doc
SA/SARE/URIBL/SURBL -- Ninja
12:04am up 39 days, 21:24, 16 users, load average: 0.73, 0.81, 0.57
SARE HQ http://www.rulesemporium.com/
Re: Nightly mass-checks
Posted by Theo Van Dinter <fe...@apache.org>.
On Wed, Mar 01, 2006 at 11:50:58PM -0600, Doc Schneider wrote:
> >>./mass-check --progress --all \
> >>ham:mbox:/home/masschecker/mail/ham \
> >>spam:mbox:/home/masschecker/mail/spam
> >>
> >>is what I'm personally using. The machine is a dual 500 with a gig of
> >>RAM. And perl 5.8.6 on it. Anyone have any ideas?
> Total size of the files? or how many messages in each mbox?
>
> A bit over 100k messages total
> spams:63731 hams:39385 give or take.
Hrm. The first thing that comes to mind is run with "-n -j 2". The -n part
won't speed up a lot, the -j 2 will tell mass-check to use both processors.
I'd also run with --progress so you can see how far it's gotten. 500MHz isn't
super fast unfortunately. You may have to trim down the number of messages
you're planning to process.
--
Randomly Generated Tagline:
God created anchovies; Satan put them on pizza.
Re: Nightly mass-checks
Posted by Warren Togami <wt...@redhat.com>.
Doc Schneider wrote:
> Had to change --after '3 months' to --after '-3 months'
>
> Ever have one of *those* days all week? ::thud::
>
I'm guessing --after has no way of truly handling mail incorrected dated
in the past and future? For this reason I sort my SPAM into
month-by-month folders and use only the last 3 months.
Warren Togami
wtogami@redhat.com
Re: Nightly mass-checks
Posted by Doc Schneider <ma...@maddoc.net>.
D
>> Here's what I'm trying now:
>>
>> ./mass-check --progress -n -j 2 --after '3 months' \
>> ham:mbox:/home/masschecker/mail/ham \
>> spam:mbox:/home/masschecker/mail/spam
>
> ::sigh:: This is what happened.
>
> status: starting scan stage now: 2006-03-02
> 00:18:26
> archive-iterator: no messages to process
>
> Any ideas?
>
Had to change --after '3 months' to --after '-3 months'
Ever have one of *those* days all week? ::thud::
--
-Doc
SA/SARE -- Ninja
1:28am up 39 days, 22:48, 16 users, load average: 0.40, 0.42, 0.48
SARE HQ http://www.rulesemporium.com/
Re: Nightly mass-checks
Posted by Doc Schneider <ma...@maddoc.net>.
Doc Schneider wrote:
> Doc Schneider wrote:
>> Duncan Findlay wrote:
>>> On Wed, Mar 01, 2006 at 11:50:58PM -0600, Doc Schneider wrote:
>>>>>> is what I'm personally using. The machine is a dual 500 with a gig
>>>>>> of RAM. And perl 5.8.6 on it. Anyone have any ideas?
>>>>> What size are these mailboxes?
>>>>>
>>>> Total size of the files? or how many messages in each mbox?
>>>>
>>>> Size of ham is 333 megs
>>>> Size of spam is 535 megs.
>>>>
>>>> A bit over 100k messages total
>>>> spams:63731 hams:39385 give or take.
>>>
>>> My current checks are roughly 40k messages. I use --after '6 months'
>>> on a much larger corpus, but that gets me down to 40k. (On my weekly
>>> net-enabled runs I use --after '1 month')
>>>
>>> It takes roughly 3 hours start to finish (including scanning the
>>> corpus and rsyncing), this is on a 2.8 GHz P4 w/ 1GB RAM.
>>>
>>> Suggestions:
>>>
>>> - Get rid of --all; you could be hitting some giant messages and
>>> burning a lot of CPU.
>>> - Use -j2 since you have 2 processors... might as well use them.
>>> - Trim your corpus, (use --after)
>>>
>>
>> Trouble is most of my corpus is less than 6 months old. Should I try
>> --after '3 months' ??
>>
>> Going to get rid of --all too. and use the -j 2 to see if that speeds
>> things up some.
>>
>
>
> Here's what I'm trying now:
>
> ./mass-check --progress -n -j 2 --after '3 months' \
> ham:mbox:/home/masschecker/mail/ham \
> spam:mbox:/home/masschecker/mail/spam
::sigh:: This is what happened.
status: starting scan stage now: 2006-03-02
00:18:26
archive-iterator: no messages to process
Any ideas?
--
-Doc
SA/SARE -- Ninja
12:40am up 39 days, 22:00, 16 users, load average: 0.46, 0.43, 0.42
SARE HQ http://www.rulesemporium.com/
Re: Nightly mass-checks
Posted by Doc Schneider <ma...@maddoc.net>.
Doc Schneider wrote:
> Duncan Findlay wrote:
>> On Wed, Mar 01, 2006 at 11:50:58PM -0600, Doc Schneider wrote:
>>>>> is what I'm personally using. The machine is a dual 500 with a gig
>>>>> of RAM. And perl 5.8.6 on it. Anyone have any ideas?
>>>> What size are these mailboxes?
>>>>
>>> Total size of the files? or how many messages in each mbox?
>>>
>>> Size of ham is 333 megs
>>> Size of spam is 535 megs.
>>>
>>> A bit over 100k messages total
>>> spams:63731 hams:39385 give or take.
>>
>> My current checks are roughly 40k messages. I use --after '6 months'
>> on a much larger corpus, but that gets me down to 40k. (On my weekly
>> net-enabled runs I use --after '1 month')
>>
>> It takes roughly 3 hours start to finish (including scanning the
>> corpus and rsyncing), this is on a 2.8 GHz P4 w/ 1GB RAM.
>>
>> Suggestions:
>>
>> - Get rid of --all; you could be hitting some giant messages and
>> burning a lot of CPU.
>> - Use -j2 since you have 2 processors... might as well use them.
>> - Trim your corpus, (use --after)
>>
>
> Trouble is most of my corpus is less than 6 months old. Should I try
> --after '3 months' ??
>
> Going to get rid of --all too. and use the -j 2 to see if that speeds
> things up some.
>
Here's what I'm trying now:
./mass-check --progress -n -j 2 --after '3 months' \
ham:mbox:/home/masschecker/mail/ham \
spam:mbox:/home/masschecker/mail/spam
--
-Doc
Penguins: Do it on the ice.
12:16am up 39 days, 21:36, 16 users, load average: 0.23, 0.42, 0.49
SARE HQ http://www.rulesemporium.com/
Re: Nightly mass-checks
Posted by Doc Schneider <ma...@maddoc.net>.
Duncan Findlay wrote:
> On Wed, Mar 01, 2006 at 11:50:58PM -0600, Doc Schneider wrote:
>>>> is what I'm personally using. The machine is a dual 500 with a gig of
>>>> RAM. And perl 5.8.6 on it. Anyone have any ideas?
>>> What size are these mailboxes?
>>>
>> Total size of the files? or how many messages in each mbox?
>>
>> Size of ham is 333 megs
>> Size of spam is 535 megs.
>>
>> A bit over 100k messages total
>> spams:63731 hams:39385 give or take.
>
> My current checks are roughly 40k messages. I use --after '6 months'
> on a much larger corpus, but that gets me down to 40k. (On my weekly
> net-enabled runs I use --after '1 month')
>
> It takes roughly 3 hours start to finish (including scanning the
> corpus and rsyncing), this is on a 2.8 GHz P4 w/ 1GB RAM.
>
> Suggestions:
>
> - Get rid of --all; you could be hitting some giant messages and burning a lot of CPU.
> - Use -j2 since you have 2 processors... might as well use them.
> - Trim your corpus, (use --after)
>
Trouble is most of my corpus is less than 6 months old. Should I try
--after '3 months' ??
Going to get rid of --all too. and use the -j 2 to see if that speeds
things up some.
--
-Doc
SA/SARE -- Ninja
12:08am up 39 days, 21:28, 16 users, load average: 0.39, 0.68, 0.57
SARE HQ http://www.rulesemporium.com/
Re: Nightly mass-checks
Posted by Duncan Findlay <du...@debian.org>.
On Wed, Mar 01, 2006 at 11:50:58PM -0600, Doc Schneider wrote:
> >>is what I'm personally using. The machine is a dual 500 with a gig of
> >>RAM. And perl 5.8.6 on it. Anyone have any ideas?
> >
> >What size are these mailboxes?
> >
> Total size of the files? or how many messages in each mbox?
>
> Size of ham is 333 megs
> Size of spam is 535 megs.
>
> A bit over 100k messages total
> spams:63731 hams:39385 give or take.
My current checks are roughly 40k messages. I use --after '6 months'
on a much larger corpus, but that gets me down to 40k. (On my weekly
net-enabled runs I use --after '1 month')
It takes roughly 3 hours start to finish (including scanning the
corpus and rsyncing), this is on a 2.8 GHz P4 w/ 1GB RAM.
Suggestions:
- Get rid of --all; you could be hitting some giant messages and burning a lot of CPU.
- Use -j2 since you have 2 processors... might as well use them.
- Trim your corpus, (use --after)
--
Duncan Findlay
Re: Nightly mass-checks
Posted by Doc Schneider <ma...@maddoc.net>.
Duncan Findlay wrote:
> On Wed, Mar 01, 2006 at 10:57:27PM -0600, Doc Schneider wrote:
>> I started one with an rsync'd version I grabbed last night about this
>> time and it is still going. Says it is 50% complete. I think I'm missing
>> an option or something.
>> ./mass-check --progress --all \
>> ham:mbox:/home/masschecker/mail/ham \
>> spam:mbox:/home/masschecker/mail/spam
>>
>> is what I'm personally using. The machine is a dual 500 with a gig of
>> RAM. And perl 5.8.6 on it. Anyone have any ideas?
>
> What size are these mailboxes?
>
Total size of the files? or how many messages in each mbox?
Size of ham is 333 megs
Size of spam is 535 megs.
A bit over 100k messages total
spams:63731 hams:39385 give or take.
--
-Doc
Penguins: Do it on the ice.
11:44pm up 39 days, 21:04, 16 users, load average: 0.11, 0.28, 0.33
SARE HQ http://www.rulesemporium.com/
Re: Nightly mass-checks
Posted by Duncan Findlay <du...@debian.org>.
On Wed, Mar 01, 2006 at 10:57:27PM -0600, Doc Schneider wrote:
> I started one with an rsync'd version I grabbed last night about this
> time and it is still going. Says it is 50% complete. I think I'm missing
> an option or something.
> ./mass-check --progress --all \
> ham:mbox:/home/masschecker/mail/ham \
> spam:mbox:/home/masschecker/mail/spam
>
> is what I'm personally using. The machine is a dual 500 with a gig of
> RAM. And perl 5.8.6 on it. Anyone have any ideas?
What size are these mailboxes?
--
Duncan Findlay
Re: Nightly mass-checks
Posted by Doc Schneider <ma...@maddoc.net>.
Theo Van Dinter wrote:
>> 4) These is no 4
>
> No poofters?
>
Ah. 4) expected length of time for nightly mass-check to complete.
I started one with an rsync'd version I grabbed last night about this
time and it is still going. Says it is 50% complete. I think I'm missing
an option or something.
./mass-check --progress --all \
ham:mbox:/home/masschecker/mail/ham \
spam:mbox:/home/masschecker/mail/spam
is what I'm personally using. The machine is a dual 500 with a gig of
RAM. And perl 5.8.6 on it. Anyone have any ideas?
--
-Doc
Penguins: Do it on the ice.
10:52pm up 39 days, 20:12, 16 users, load average: 0.47, 0.73, 0.56
SARE HQ http://www.rulesemporium.com/
Re: Nightly mass-checks
Posted by Theo Van Dinter <fe...@apache.org>.
On Tue, Feb 28, 2006 at 10:29:10PM -0600, Doc Schneider wrote:
> 1) Command line to run a mass-check $./mass-check --all --progress
> ham:mbox:path/to/ham (does this need a / or a /* to run through a whole
> directory of mbox ham files? Ditto with spam?)
You can specify a wildcard if you want to, but it's more efficient to
aim at the directory of ham files, and mass-check will go through it
non-recursively. ie:
mass-check ham:mbox:path/to/dir
runs through all the mbox files in dir but not directories underneath.
> 3) A good example of how to do a mass-check using the SVN method. (Theo
> gave me some clues on using it so am pretty much set for doing this.)
You can check the contrib/run-corpora script. It does a few other things,
but the core does the SVN updates, then runs run-masses which essentially
just runs mass-check with some options.
> 4) These is no 4
No poofters?
--
Randomly Generated Tagline:
A formal parsing algorithm should not always be used.
-- D. Gries
Re: Nightly mass-checks
Posted by Duncan Findlay <du...@debian.org>.
On Tue, Feb 28, 2006 at 10:29:10PM -0600, Doc Schneider wrote:
> Gang,
>
> I'm working on a better wiki page for nightly mass-checks and an easier
> way for those old and new to SA to be able to do these.
>
> A few things which I'm not entirely sure of are:
>
> 1) Command line to run a mass-check $./mass-check --all --progress
> ham:mbox:path/to/ham (does this need a / or a /* to run through a whole
> directory of mbox ham files? Ditto with spam?)
I recommend using mass-check's -f option, which lets you define all of
your targets (ham:dir:/path, etc) in one file, and just pass the name
of the file to mass-check.
For nightly checks I do:
mass-check -f <filename> --after='6 months'
For weekly checks I do:
mass-check --net -j 8 -f <filename> --after='6 months'
Looks like I don't use --all, I probably should. --progress is
probably unnecessary if you're running this from a cron job.
> 2) Command line for maildir files? ham:dir:/path/to/ham (Wondering the
> same on this also about needing a / or /*)
For maildirs:
ham:dir:/path/to/maildir/cur
For a dir of mboxes:
ham:dir/path/to/mboxdir/*
> 3) A good example of how to do a mass-check using the SVN method. (Theo
> gave me some clues on using it so am pretty much set for doing this.)
It seems this is the "old", almost deprecated way now... Guess I
missed that memo :-)
Manually:
Check out a spamassassin tree
Nightly:
wget http://rsync.spamassassin.org/weekly-versions.txt or nightly-versions.txt
take the revision number from the last line of that file
(inside a loop with some (randomized) retry logic)
svn update -r <revision number>
perl Makefile.PL
make
run mass-check
upload results via rsync (setting the RSYNC_PASSWORD environment variable is useful)
--
My script is designed to be run every hour, and it will only run if it
hasn't yet run "today" where "today" is defined as since 9:00 UTC
(when the revision file gets updated)
> 5) Anyone having a particular problem that should be addressed for doing
> mass-checks?
Now that I think about it, with the rules sandbox stuff, the SVN
checkout probably won't always get the same revision if people update
their SVN at different times of the day. Is this a bad thing.
I just tested this. If I run my nightly mass checks at 11:00 UTC (for
example) and someone commits a rule change between 9:00 UTC and 11:00
UTC, I'll pick up the rule and test it, but others might not. I don't
know what ramifications this might happen.
duncf@gold:~/svn/spamassassin-nightly$ svn update -r 381595
Fetching external item into 'rulesrc'
U rulesrc/sandbox/duncf/20_drugs.cf
Updated external to revision 381910.
Updated to revision 381595.
Note the rules directory gets updated to 381910 instead of 381595.
Interesting...
--
Duncan Findlay