You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spamassassin.apache.org by Theo Van Dinter <fe...@apache.org> on 2007/01/30 04:59:34 UTC
Using SQL for scoregen run is slow as heck!
I'm mostly just venting, but I have 2 machines trying to do the
scoregen run, so I need to share the Bayes DB, and chose SQL. In total
across those machines, I'm currently only seeing ~25 msgs/min or 0.5
messages/sec. So for my ~215k messages, that's 430k seconds, or 5 days
of processing. Set 1 would take ~12h for this same set of messages,
so bayes adds a 10x time penalty?
Am I missing something? MySQL w/ InnoDB + and BayesStore::MySQL, MySQL 4.1.
Granted, I haven't timed this against SDBM or DBM, but I'll take any hints
for MySQL tuning. An upside of using the mass-check client/server code
is the ability to stop the clients, change MySQL configs, and restart
the clients, and not lose the previous work. :)
--
Randomly Selected Tagline:
"The very powerful and the very stupid have one thing in common.
Instead of altering their views to fit the facts, they alter the facts
to fit their views ... which can be very uncomfortable if you happen to
be one of the facts that needs altering." - Doctor Who, "Face of Evil"
Re: Using SQL for scoregen run is slow as heck!
Posted by Theo Van Dinter <fe...@apache.org>.
On Mon, Jan 29, 2007 at 11:21:20PM -0500, Daryl C. W. O'Shea wrote:
> MySQL isn't hitting swap perhaps? That's the only way I've seen it go
> that slow (with spamd) and I've done zero tuning of MySQL.
Nope, no swap usage on the machine.
> Is the load on the machine running the DB actually high, or is there an
> issue somewhere else... like a crappy network switch?
The load's currently hovering between 1 and 2 w/ dual CPUs, so that's
fine. The network connection between the two machines is a crossover
cable w/ a GbE connection, so that's working fine.
I noticed that if I change the local machine from using that interface
to using localhost via dsn, I get a 10x speed increase, which is strange. It
appears that dsn=ethernet-IP goes through the network stack. dsn=localhost
goes to the socket.
So now it's ~150 messages/min, or 2.5 messages/sec.
But the extra machine isn't really doing anything for me, since the
single machine with SDBM is doing ~2.4 messages/sec.
--
Randomly Selected Tagline:
Anyone who considers protocol unimportant has never dealt with a cat.
-- R. Heinlein
Re: Using SQL for scoregen run is slow as heck!
Posted by "Daryl C. W. O'Shea" <sp...@dostech.ca>.
Theo Van Dinter wrote:
> I'm mostly just venting, but I have 2 machines trying to do the
> scoregen run, so I need to share the Bayes DB, and chose SQL. In total
> across those machines, I'm currently only seeing ~25 msgs/min or 0.5
> messages/sec. So for my ~215k messages, that's 430k seconds, or 5 days
> of processing. Set 1 would take ~12h for this same set of messages,
> so bayes adds a 10x time penalty?
>
> Am I missing something? MySQL w/ InnoDB + and BayesStore::MySQL, MySQL 4.1.
>
>
> Granted, I haven't timed this against SDBM or DBM, but I'll take any hints
> for MySQL tuning. An upside of using the mass-check client/server code
> is the ability to stop the clients, change MySQL configs, and restart
> the clients, and not lose the previous work. :)
MySQL isn't hitting swap perhaps? That's the only way I've seen it go
that slow (with spamd) and I've done zero tuning of MySQL.
Is the load on the machine running the DB actually high, or is there an
issue somewhere else... like a crappy network switch?
For comparison, I'm getting 9.5 messages/second with 8 processes on my
2.8 Ghz 4MB Pentium duo core, with software sata raid (LVM), using SDBM.
Daryl
Re: Using SQL for scoregen run is slow as heck!
Posted by Doc Schneider <ma...@maddoc.net>.
Theo Van Dinter wrote:
> On Tue, Jan 30, 2007 at 03:50:20PM +0000, Justin Mason wrote:
>> It *would* be cool to log that in the --progress messages though... ;)
>
> Sure, open a ticket about it. Something for 3.3. :)
>
Appears mine is going about 60/min or 1 a sec. Geesh. I've got like over
250k total messages... this is going to take a while. ::sigh::
--
-Doc
SA/SARE/URIBL/SURBL -- Ninja
10:24am up 16 days, 19:22, 15 users, load average: 4.13, 2.67, 2.03
SARE HQ http://www.rulesemporium.com/
Re: Using SQL for scoregen run is slow as heck!
Posted by Theo Van Dinter <fe...@apache.org>.
On Tue, Jan 30, 2007 at 03:50:20PM +0000, Justin Mason wrote:
> It *would* be cool to log that in the --progress messages though... ;)
Sure, open a ticket about it. Something for 3.3. :)
--
Randomly Selected Tagline:
"You only use spherical coordinates when you've got a sphere." - Prof. Farr
Re: Using SQL for scoregen run is slow as heck!
Posted by Theo Van Dinter <fe...@apache.org>.
On Tue, Jan 30, 2007 at 08:30:09AM -0600, Doc Schneider wrote:
> I didn't know it was possible to use MySQL for the
> scoregen runs. I would have used it instead. I also should have used a
> smaller corpus.
Yeah, you can do whatever you want for the scoregen run, as long as it doesn't
change the results.
> How do you see how many msgs/min it is processing? Be aware I'm running
> on my first cuppa coffee. 8*)
:) I do it in a really simple way:
while true; do date ; wc -l *.log; echo; sleep 60; done
--
Randomly Selected Tagline:
"A journey of a thousand miles begins with but a step." - Chinese Proverb
Re: Using SQL for scoregen run is slow as heck!
Posted by Doc Schneider <ma...@maddoc.net>.
Theo Van Dinter wrote:
> I'm mostly just venting, but I have 2 machines trying to do the
> scoregen run, so I need to share the Bayes DB, and chose SQL. In total
> across those machines, I'm currently only seeing ~25 msgs/min or 0.5
> messages/sec. So for my ~215k messages, that's 430k seconds, or 5 days
> of processing. Set 1 would take ~12h for this same set of messages,
> so bayes adds a 10x time penalty?
>
> Am I missing something? MySQL w/ InnoDB + and BayesStore::MySQL, MySQL 4.1.
>
>
> Granted, I haven't timed this against SDBM or DBM, but I'll take any hints
> for MySQL tuning. An upside of using the mass-check client/server code
> is the ability to stop the clients, change MySQL configs, and restart
> the clients, and not lose the previous work. :)
>
Thia all sounds a lot like mine which is doing the bayes through SDBM.
It is now just over 50% done and looking like it is going to run for a
couple more days. I didn't know it was possible to use MySQL for the
scoregen runs. I would have used it instead. I also should have used a
smaller corpus.
The machine I am running mine on is a 1.8Ghz with 1.5Gigs of RAM. Only
seems to swap when some of the heavier cron events run. I should have
used a different machine.
How do you see how many msgs/min it is processing? Be aware I'm running
on my first cuppa coffee. 8*)
--
-Doc
SA/SARE/URIBL/SURBL -- Ninja
8:20am up 16 days, 17:18, 15 users, load average: 3.73, 2.45, 2.19
SARE HQ http://www.rulesemporium.com/