You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Ben Poliakoff <be...@reed.edu> on 2005/02/23 20:45:47 UTC

bayes_expiry_max_db_size setting for sitewide installation?

What sort of guidelines/rules of thumb/formulas have people used to
determine the bayes_expiry_max_db_size setting for a sitewide bayes
database?

The Mail::SpamAssassin::Conf man page says the default is 150000 tokens
(which, it says, is equivalent to roughly 8mb).  It seems a little
extreme to simply multiply that number by the number of users on the
server.

    8Mb * 2000 users = ~16Gb!

I'm planning on hosting this db in mysql (an SQL based bayes seems
better suited than the default "file based" option for a sitewide DB),
but clearly 16Gb is just too big....  Presumably something smaller would
work well enough, but how small is too small and how big is too big?

The only advice I've found in the list archives is:

http://marc.theaimsgroup.com/?l=spamassassin-users&m=109033803207027&w=2

    > How big are your bayes_* files on disk?  I would say personally
    > that a single-user set of Bayes files shouldn't be much more than
    > 8-10M total; a medium-size site Bayes should be ~40M _toks +
    > whatever _seen takes up; and a large sitewide Bayes may run up to
    > ~100M.  I wouldn't go much higher due to the IO/memory/filesystem
    > cache load.

Thanks!

Ben

Re: bayes_expiry_max_db_size setting for sitewide installation?

Posted by Matt Kettler <mk...@evi-inc.com>.
At 02:45 PM 2/23/2005, Ben Poliakoff wrote:
>What sort of guidelines/rules of thumb/formulas have people used to
>determine the bayes_expiry_max_db_size setting for a sitewide bayes
>database?

Personally, I'm using 200,000 on a 100 user site-wide setup. This means my 
bayes DB varies between 200k and 150ktokens (75% of max) instead of the 
default 150k-100k.

I could easily double that, but thus far have not seen much need. Being a 
business most of the email here is to users with more-or-less similar email 
patterns. If your users are widely varied in interests on the ham side you 
might want to expand it further.


>     > How big are your bayes_* files on disk?  I would say personally
>     > that a single-user set of Bayes files shouldn't be much more than
>     > 8-10M total; a medium-size site Bayes should be ~40M _toks +
>     > whatever _seen takes up; and a large sitewide Bayes may run up to
>     > ~100M.  I wouldn't go much higher due to the IO/memory/filesystem
>     > cache load.

That seems like reasonable advice.. if 8MB is 150,000 tokens, 100MB would 
be about 1,875,000 tokens.. That's probably a good upper bound..

Looking around on the web, I see most quoted configs being between 1.5M and 
100k tokens. (excluding the people trying to make it less than 100k, which 
won't ever work).

I also saw one guide with 20M in their config, but I'm wondering if the 
guide author typoed or is confused and thinks it's specified in bytes 
instead of tokens. That bayes DB would be 1GB..
http://www.postfix-howto.de/spamassassin_conf.html

With 2000 users I might consider starting with something in the middle, 
about 800k or so, and see how it goes..



Re: bayes_expiry_max_db_size setting for sitewide installation?

Posted by Michael Parker <pa...@pobox.com>.
On Wed, Feb 23, 2005 at 03:31:33PM -0800, Ben Poliakoff wrote:
> Very useful comments all around.  I now have some reasonable numbers
> (tokens and overall db size) to kick around.  I also will reconsider
> using a "dbx" backed bayes.  

FYI, I've found that at least with MySQL, the db size ends up being
smaller than the DBM based databases.  I've forgotten the percentage,
but you can probably find it somewhere on the users or dev list where
I mentioned it once upon a time.

Don't quote me, but I want to say 7-10% smaller, but really I've
forgotten what the actual number was.

Michael

Re: bayes_expiry_max_db_size setting for sitewide installation?

Posted by Ben Poliakoff <be...@reed.edu>.
* Ben Poliakoff <be...@reed.edu> [20050223 11:46]:
> What sort of guidelines/rules of thumb/formulas have people used to
> determine the bayes_expiry_max_db_size setting for a sitewide bayes
> database?

Thanks Matt, Kris, and Kai,

Very useful comments all around.  I now have some reasonable numbers
(tokens and overall db size) to kick around.  I also will reconsider
using a "dbx" backed bayes.  

Ben

Re: bayes_expiry_max_db_size setting for sitewide installation?

Posted by Kris Deugau <kd...@vianet.ca>.
Ben Poliakoff wrote:
> What sort of guidelines/rules of thumb/formulas have people used to
> determine the bayes_expiry_max_db_size setting for a sitewide bayes
> database?

Modify until it feels right.  <g>

> The Mail::SpamAssassin::Conf man page says the default is 150000
> tokens (which, it says, is equivalent to roughly 8mb).  It seems a
> little extreme to simply multiply that number by the number of users
> on the server.
>     8Mb * 2000 users = ~16Gb!

Definitely extreme.  Keep in mind that on a global Bayes DB, there will
be a LOT of similar/identical tokens comparing per-user mail flow with
systemwide information.  Even in an ISP environment.

> I'm planning on hosting this db in mysql (an SQL based bayes seems
> better suited than the default "file based" option for a sitewide
> DB),

*shrug*  I've been running with a file-based global Bayes for several
years on a number of systems now.  Other than inherent "not enough
CPU/memory/disk/IO" problems with one system generally, I've had zero
trouble.  (Bayes, in and of itself, is a pretty trival load on that
system AFAICS.)

> The only advice I've found in the list archives is:
>http://marc.theaimsgroup.com/?l=spamassassin-users&m=109033803207027&w=2
>     > How big are your bayes_* files on disk?  I would say personally
>     > that a single-user set of Bayes files shouldn't be much more
>     > than 8-10M total; a medium-size site Bayes should be ~40M _toks
>     > + whatever _seen takes up; and a large sitewide Bayes may run
>     > up to ~100M.  I wouldn't go much higher due to the
>     > IO/memory/filesystem cache load.

That's probably about the best specific advice you'll get.  ;)

I'd start by trying 1M tokens (~40M in my setup), and see how the mail
flow looks.  If Bayes seems to be getting into trouble, bump it up to
2M, and so on.  Smaller is better for I/O, but bigger is better for
accuracy.  Exactly where the sweet spot is will vary depending on your
mail flow and your particular server setup.

-kgd
-- 
Get your mouse off of there!  You don't know where that email has been!

Re: bayes_expiry_max_db_size setting for sitewide installation?

Posted by Kai Schaetzl <ma...@conactive.com>.
Ben Poliakoff wrote on Wed, 23 Feb 2005 11:45:47 -0800:

> 8Mb * 2000 users = ~16Gb!
>

nonono, a site-wide install doesn't scale per user, one fits all. We have 
between 0,5 - 1 mio tokens and up to 40 MB databases for setups between a 
dozen and a thousand users. Works well. dbx, not SQL. Not sure if Bayes 
gives any advantage other than you can easier expire it manually if the SA 
expiry has a problem.

Kai

-- 
Kai Schätzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com
IE-Center: http://ie5.de & http://msie.winware.org