You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by redtailjason <ja...@redtailtechnology.com> on 2014/08/20 15:15:18 UTC
Delays with Check_Bayes
Hello and good morning. We are running into some delays that we are trying to
pin down a root cause for.
Below are some examples. Within the examples, you can see that the
check_bayes: scan is consuming most of the timing. Does anyone have any
suggests on what to look at? We use 3.3.2. We have eight scanners setup to
handle the scanning with 5GB RAM and 4 CPUs each. Volume is 250K - 500K per
day.
Aug 19 5:06:07 amavis[1581]: (01581-07-2) TIMING-SA total 138564 ms - parse:
2 0.00% extract_message_metadata: 37 (0.0%) get_uri_detail_list: 7 (0.0%)
tests_pri_-1000: 13 (0.0%) tests_pri_-950: 1.08 (0.0%) tests_pri_-900: 1.13
(0.0%) tests_pri_-400: 137793 (99.4%) check_bayes: 137786 (99.4%)
tests_pri_0: 708 (0.5%) check_dkim_adsp: 15 (0.0%) check_spf: 10 (0.0%)
poll_dns_idle: 6 (0.0%) tests_pri_500: 3 (0.0%) get_report: 0.88 (0.0%)
Aug 19 6:06:08 amavis[1271]: (01271-12-3) TIMING-SA total 118903 ms - parse:
1.75 0.00% extract_message_metadata: 34 (0.0%) get_uri_detail_list: 8 (0.0%)
tests_pri_-1000: 7 (0.0%) tests_pri_-950: 1.11 (0.0%) tests_pri_-900: 1.18
(0.0%) tests_pri_-400: 118273 (99.5%) check_bayes: 118266 (99.5%)
tests_pri_0: 419 (0.4%) check_dkim_adsp: 46 (0.0%) check_spf: 5 (0.0%)
poll_dns_idle: 152 (0.1%) tests_pri_500: 156 (0.1%) get_report: 5 (0.0%)
Aug 19 5:06:21 amavis[6764]: (06764-02-6) TIMING-SA total 99680 ms - parse:
2 0.00% extract_message_metadata: 37 (0.0%) get_uri_detail_list: 7 (0.0%)
tests_pri_-1000: 13 (0.0%) tests_pri_-950: 1.08 (0.0%) tests_pri_-900: 1.13
(0.0%) tests_pri_-400: 98881 (99.2%) check_bayes: 98874 (99.2%) tests_pri_0:
736 (0.7%) check_dkim_adsp: 12 (0.0%) check_spf: 5 (0.0%) poll_dns_idle:
1.26 (0.0%) tests_pri_500: 3 (0.0%) get_report: 0.85 (0.0%)
Aug 19 5:06:19 amavis[9621]: (09621-13-6) TIMING-SA total 99636 ms - parse:
2 0.00% extract_message_metadata: 38 (0.0%) get_uri_detail_list: 7 (0.0%)
tests_pri_-1000: 12 (0.0%) tests_pri_-950: 1.09 (0.0%) tests_pri_-900: 1.14
(0.0%) tests_pri_-400: 98847 (99.2%) check_bayes: 98839 (99.2%) tests_pri_0:
726 (0.7%) check_dkim_adsp: 11 (0.0%) check_spf: 6 (0.0%) poll_dns_idle: 2
(0.0%) tests_pri_500: 3 (0.0%) get_report: 0.87 (0.0%)
Aug 19 6:06:07 amavis[16447]: (16447-06-10) TIMING-SA total 90079 ms -
parse: 2 0.00% extract_message_metadata: 34 (0.0%) get_uri_detail_list: 5
(0.0%) tests_pri_-1000: 9 (0.0%) tests_pri_-950: 1.24 (0.0%) tests_pri_-900:
1.35 (0.0%) tests_pri_-400: 89698 (99.6%) check_bayes: 89685 (99.6%)
tests_pri_0: 323 (0.4%) check_dkim_adsp: 51 (0.1%) check_spf: 20 (0.0%)
poll_dns_idle: 16 (0.0%) tests_pri_500: 3 (0.0%) get_report: 1.03 (0.0%)
Aug 19 5:07:28 amavis[3901]: (03901-02-7) TIMING-SA total 87855 ms - parse:
4 0.00% extract_message_metadata: 84 (0.1%) get_uri_detail_list: 22 (0.0%)
tests_pri_-1000: 32 (0.0%) tests_pri_-950: 1.10 (0.0%) tests_pri_-900: 1.18
(0.0%) tests_pri_-400: 87015 (99.0%) check_bayes: 86980 (99.0%) tests_pri_0:
699 (0.8%) check_dkim_adsp: 32 (0.0%) check_spf: 9 (0.0%) poll_dns_idle:
1.03 (0.0%) tests_pri_500: 4 (0.0%) get_report: 1.03 (0.0%)
Aug 19 6:15:03 amavis[7851]: (07851-02-3) TIMING-SA total 81154 ms - parse:
2 0.00% extract_message_metadata: 41 (0.1%) get_uri_detail_list: 11 (0.0%)
tests_pri_-1000: 11 (0.0%) tests_pri_-950: 1.05 (0.0%) tests_pri_-900: 1.11
(0.0%) tests_pri_-400: 80789 (99.5%) check_bayes: 80778 (99.5%) tests_pri_0:
299 (0.4%) check_spf: 6 (0.0%) poll_dns_idle: 1.31 (0.0%) tests_pri_500: 3
(0.0%) get_report: 1.14 (0.0%)
--
View this message in context: http://spamassassin.1065346.n5.nabble.com/Delays-with-Check-Bayes-tp111067.html
Sent from the SpamAssassin - Users mailing list archive at Nabble.com.
Re: Delays with Check_Bayes
Posted by Jeremy McSpadden <je...@fluxlabs.net>.
Do not have enough HAM to kick on bayes.
--
Jeremy McSpadden
Flux Labs | http://www.fluxlabs.net<http://www.fluxlabs.net/> | Endless Solutions
Office : 850-250-5590x501<tel:850-250-5590;501> | Cell : 850-890-2543<tel:850-890-2543> | Fax : 850-254-2955<tel:850-254-2955>
On Aug 20, 2014, at 10:36 AM, "redtailjason" <ja...@redtailtechnology.com>> wrote:
Aug 20 07:54:54.456 [6955] dbg: bayes: not available for scanning, only 0
ham(s) in bayes DB < 200
Re: Delays with Check_Bayes
Posted by redtailjason <ja...@redtailtechnology.com>.
Thank you for the clarification, sorry about that. Below is the updated
information:
sa-learn --dump magic
0.000 0 3 0 non-token data: bayes db version
0.000 0 2687726 0 non-token data: nspam
0.000 0 846578 0 non-token data: nham
0.000 0 241756 0 non-token data: ntokens
0.000 0 1357225630 0 non-token data: oldest atime
0.000 0 1408558344 0 non-token data: newest atime
0.000 0 0 0 non-token data: last journal sync
atime
0.000 0 1408541108 0 non-token data: last expiry atime
0.000 0 43200 0 non-token data: last expire atime
delta
0.000 0 0 0 non-token data: last expire
reduction count
$ spamassassin --lint -D bayes
Aug 20 11:13:37.432 [16980] warn: netset: cannot include 127.0.0.1/32 as it
has already been included
Aug 20 11:13:37.978 [16980] dbg: bayes: learner_new
self=Mail::SpamAssassin::Plugin::Bayes=HASH(0x3e11a90),
bayes_store_module=Mail::SpamAssassin::BayesStore::MySQL
Aug 20 11:13:37.990 [16980] dbg: bayes: using username: amavis
Aug 20 11:13:37.990 [16980] dbg: bayes: learner_new: got
store=Mail::SpamAssassin::BayesStore::MySQL=HASH(0x40f2978)
Aug 20 11:13:38.011 [16980] dbg: bayes: database connection established
Aug 20 11:13:38.012 [16980] dbg: bayes: found bayes db version 3
Aug 20 11:13:38.015 [16980] dbg: bayes: Using userid: 1
Aug 20 11:13:38.040 [16980] dbg: bayes: corpus size: nspam = 2687726, nham =
846578
Aug 20 11:13:38.042 [16980] dbg: bayes: header tokens for *F = "U*ignore
D*compiling.spamassassin.taint.org D*spamassassin.taint.org D*taint.org
D*org"
Aug 20 11:13:38.042 [16980] dbg: bayes: header tokens for *m = " 1408558416
lint_rules "
Aug 20 11:13:38.042 [16980] dbg: bayes: header tokens for
x-spam-relays-external = " "
Aug 20 11:13:38.042 [16980] dbg: bayes: header tokens for
x-spam-relays-internal = " "
Aug 20 11:13:38.042 [16980] dbg: bayes: header tokens for *RT = " "
Aug 20 11:13:38.042 [16980] dbg: bayes: header tokens for *RU = " "
Aug 20 11:13:38.043 [16980] dbg: bayes: tok_get_all: token count: 20
Aug 20 11:13:38.045 [16980] dbg: bayes: cannot use bayes on this message;
not enough usable tokens found
Aug 20 11:13:38.045 [16980] dbg: bayes: not scoring message, returning undef
Please let me know if you need additional information.
--
View this message in context: http://spamassassin.1065346.n5.nabble.com/Delays-with-Check-Bayes-tp111067p111085.html
Sent from the SpamAssassin - Users mailing list archive at Nabble.com.
Re: Delays with Check_Bayes
Posted by Karsten Bräckelmann <gu...@rudersport.de>.
On Wed, 2014-08-20 at 08:51 -0700, redtailjason wrote:
> The initial post was data extracted from mail.log on the scanner using cat
> /var/log/mail.log | grep check_bayes while logged as administrator.
It doesn't matter what user greps the logs.
It was Amavis generating the logs. Thus, for debugging, all execution of
Amavis or SA commands must be done as the user Amavis runs as.
--
char *t="\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i<l;i++){ i%8? c<<=1:
(c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}
Re: Delays with Check_Bayes
Posted by redtailjason <ja...@redtailtechnology.com>.
AXB,
The initial post was data extracted from mail.log on the scanner using cat
/var/log/mail.log | grep check_bayes while logged as administrator.
--
View this message in context: http://spamassassin.1065346.n5.nabble.com/Delays-with-Check-Bayes-tp111067p111081.html
Sent from the SpamAssassin - Users mailing list archive at Nabble.com.
Re: Delays with Check_Bayes
Posted by Axb <ax...@gmail.com>.
On 08/20/2014 05:35 PM, redtailjason wrote:
> Below is the output that you are seeking:
>
> $ spamassassin --lint -D bayes
> Aug 20 07:54:53.816 [6955] warn: netset: cannot include 127.0.0.1/32 as it
> has already been included
> Aug 20 07:54:54.415 [6955] dbg: bayes: learner_new
> self=Mail::SpamAssassin::Plugin::Bayes=HASH(0x382ddd0),
> bayes_store_module=Mail::SpamAssassin::BayesStore::MySQL
> Aug 20 07:54:54.428 [6955] dbg: bayes: using username: administrator
> Aug 20 07:54:54.428 [6955] dbg: bayes: learner_new: got
> store=Mail::SpamAssassin::BayesStore::MySQL=HASH(0x3b0e068)
> Aug 20 07:54:54.448 [6955] dbg: bayes: database connection established
> Aug 20 07:54:54.450 [6955] dbg: bayes: found bayes db version 3
> Aug 20 07:54:54.452 [6955] dbg: bayes: Using userid: 3
> Aug 20 07:54:54.456 [6955] dbg: bayes: not available for scanning, only 0
> ham(s) in bayes DB < 200
> Aug 20 07:54:54.464 [6955] dbg: bayes: database connection established
> Aug 20 07:54:54.464 [6955] dbg: bayes: found bayes db version 3
> Aug 20 07:54:54.467 [6955] dbg: bayes: Using userid: 3
> Aug 20 07:54:54.472 [6955] dbg: bayes: not available for scanning, only 0
> ham(s) in bayes DB < 200
>
> Please let me know if you need additional information.
you've run this a "administrator"
from your first msg, you run amavis, does you amavis also run under user
"administrator"? You should do/post debugging under that same user.
Re: Delays with Check_Bayes
Posted by Benny Pedersen <me...@junc.eu>.
On 20. aug. 2014 17.36.39 redtailjason <ja...@redtailtechnology.com> wrote:
> $ spamassassin --lint -D bayes
> Aug 20 07:54:53.816 [6955] warn: netset: cannot include 127.0.0.1/32 as it
> has already been included
rfc 1700 is already in default sa config, tanks for that to developpers :)
May miss rfc 1918 defined in lokal.cf, save time in rbl checks
perldoc Mail::SpamAssassin:::Conf
Re: Delays with Check_Bayes
Posted by redtailjason <ja...@redtailtechnology.com>.
Below is the output that you are seeking:
$ spamassassin --lint -D bayes
Aug 20 07:54:53.816 [6955] warn: netset: cannot include 127.0.0.1/32 as it
has already been included
Aug 20 07:54:54.415 [6955] dbg: bayes: learner_new
self=Mail::SpamAssassin::Plugin::Bayes=HASH(0x382ddd0),
bayes_store_module=Mail::SpamAssassin::BayesStore::MySQL
Aug 20 07:54:54.428 [6955] dbg: bayes: using username: administrator
Aug 20 07:54:54.428 [6955] dbg: bayes: learner_new: got
store=Mail::SpamAssassin::BayesStore::MySQL=HASH(0x3b0e068)
Aug 20 07:54:54.448 [6955] dbg: bayes: database connection established
Aug 20 07:54:54.450 [6955] dbg: bayes: found bayes db version 3
Aug 20 07:54:54.452 [6955] dbg: bayes: Using userid: 3
Aug 20 07:54:54.456 [6955] dbg: bayes: not available for scanning, only 0
ham(s) in bayes DB < 200
Aug 20 07:54:54.464 [6955] dbg: bayes: database connection established
Aug 20 07:54:54.464 [6955] dbg: bayes: found bayes db version 3
Aug 20 07:54:54.467 [6955] dbg: bayes: Using userid: 3
Aug 20 07:54:54.472 [6955] dbg: bayes: not available for scanning, only 0
ham(s) in bayes DB < 200
Please let me know if you need additional information.
--
View this message in context: http://spamassassin.1065346.n5.nabble.com/Delays-with-Check-Bayes-tp111067p111078.html
Sent from the SpamAssassin - Users mailing list archive at Nabble.com.
Re: Delays with Check_Bayes
Posted by Axb <ax...@gmail.com>.
On 08/20/2014 04:35 PM, redtailjason wrote:
> Thank you for your response! Our Bayes is MySQL. Currently, the expiry runs
> via cron job to run during low volume times.
>
> Here is the dump from one of the scanners:
>
> netset: cannot include 127.0.0.1/32 as it has already been included
> 0.000 0 3 0 non-token data: bayes db version
> 0.000 0 613 0 non-token data: nspam
> 0.000 0 0 0 non-token data: nham
> 0.000 0 50382 0 non-token data: ntokens
> 0.000 0 1362372138 0 non-token data: oldest atime
> 0.000 0 1396547409 0 non-token data: newest atime
> 0.000 0 0 0 non-token data: last journal sync
> atime
> 0.000 0 0 0 non-token data: last expiry atime
> 0.000 0 0 0 non-token data: last expire atime
> delta
> 0.000 0 0 0 non-token data: last expire
> reduction count
>
> Please let me know if you need additional information.
is that really your production bayes DB? So little data?
> 0.000 0 613 0 non-token data: nspam
> 0.000 0 0 0 non-token data: nham
pls post the output of
spamassassin --lint -D bayes
and make sure you do this under the same user which is used by SA
Re: Delays with Check_Bayes
Posted by John Hardin <jh...@impsec.org>.
On Wed, 20 Aug 2014, redtailjason wrote:
> Thank you for your response! Our Bayes is MySQL. Currently, the expiry runs
> via cron job to run during low volume times.
Does the MySQL log say anything suggestive?
> Here is the dump from one of the scanners:
>
> netset: cannot include 127.0.0.1/32 as it has already been included
> 0.000 0 3 0 non-token data: bayes db version
> 0.000 0 613 0 non-token data: nspam
> 0.000 0 0 0 non-token data: nham
This doesn't explain the long scan time, but you can't expect Bayes to
score at all until you learn some ham too. It needs examples of both to be
able to tell the difference.
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
Efficiency can magnify good, but it magnifies evil just as well.
So, we should not be surprised to find that modern electronic
communication magnifies stupidity as *efficiently* as it magnifies
intelligence. -- Robert A. Matern
-----------------------------------------------------------------------
4 days until the 1935th anniversary of the destruction of Pompeii
Re: Delays with Check_Bayes
Posted by Karsten Bräckelmann <gu...@rudersport.de>.
On Wed, 2014-08-20 at 07:35 -0700, redtailjason wrote:
> Here is the dump from one of the scanners:
>
> netset: cannot include 127.0.0.1/32 as it has already been included
> 0.000 0 3 0 non-token data: bayes db version
> 0.000 0 613 0 non-token data: nspam
> 0.000 0 0 0 non-token data: nham
> 0.000 0 50382 0 non-token data: ntokens
> 0.000 0 1362372138 0 non-token data: oldest atime
> 0.000 0 1396547409 0 non-token data: newest atime
That's back in April -- and obviously not a production database.
You need to run sa-update as the user SA uses during scan. In your case
that's the user Amavis uses.
--
char *t="\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i<l;i++){ i%8? c<<=1:
(c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}
Re: Delays with Check_Bayes
Posted by redtailjason <ja...@redtailtechnology.com>.
Thank you for your response! Our Bayes is MySQL. Currently, the expiry runs
via cron job to run during low volume times.
Here is the dump from one of the scanners:
netset: cannot include 127.0.0.1/32 as it has already been included
0.000 0 3 0 non-token data: bayes db version
0.000 0 613 0 non-token data: nspam
0.000 0 0 0 non-token data: nham
0.000 0 50382 0 non-token data: ntokens
0.000 0 1362372138 0 non-token data: oldest atime
0.000 0 1396547409 0 non-token data: newest atime
0.000 0 0 0 non-token data: last journal sync
atime
0.000 0 0 0 non-token data: last expiry atime
0.000 0 0 0 non-token data: last expire atime
delta
0.000 0 0 0 non-token data: last expire
reduction count
Please let me know if you need additional information.
--
View this message in context: http://spamassassin.1065346.n5.nabble.com/Delays-with-Check-Bayes-tp111067p111074.html
Sent from the SpamAssassin - Users mailing list archive at Nabble.com.
Re: Delays with Check_Bayes
Posted by Axb <ax...@gmail.com>.
On 08/20/2014 03:15 PM, redtailjason wrote:
> Hello and good morning. We are running into some delays that we are trying to
> pin down a root cause for.
>
> Below are some examples. Within the examples, you can see that the
> check_bayes: scan is consuming most of the timing. Does anyone have any
> suggests on what to look at? We use 3.3.2. We have eight scanners setup to
> handle the scanning with 5GB RAM and 4 CPUs each. Volume is
what type of Bayes backed are you using?
do you use auto expiration or via cron?
and pls post the output of
sa-learn --dump magic
Re: Delays with Check_Bayes
Posted by Karsten Bräckelmann <gu...@rudersport.de>.
On Wed, 2014-08-20 at 13:38 -0700, redtailjason wrote:
> We are seeing about 4000-7000 delayed messages per day. We do utilize a
> dedicated MySQL Server for the Bayes and all 8 scanners share it. Please let
> me know if this does not fully clarify our setup for you.
So we're talking about 1% of the messages.
Does this happen with all scanner machines, or is this isolated to a
single one? If not all scanners are affected, any differences in network
connection?
When did this start? Any relevant changes roughly about that time?
What's your DB server load? Any noticeable load spikes, like 5k times a
day? In particular, while a message is taking 2 minutes wall-clock time
for Bayes, does either the scanner or database server have an unusual
high load? Do you have MySQL logs which might show issues?
Can you reproduce the Bayes lags? That is, can you identify a sample
message, and re-process manually?
When replying, please include the relevant quoted parts you're directly
referring to. With some context it is easier to follow the thread.
--
char *t="\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i<l;i++){ i%8? c<<=1:
(c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}
Re: Delays with Check_Bayes
Posted by Karsten Bräckelmann <gu...@rudersport.de>.
On Thu, 2014-08-21 at 13:13 -0700, redtailjason wrote:
> Are you open to the possibility of upgrading to 3.4.0 and using the Redis
> backend for Bayes? (Just offering an alternative.)
>
> We have been developing and upgrade plan to 3.4. Based on this, we are
> prioritize this upgrade and will be expediting it. Thanks.
Thanks for including the part you're directly referring to, as I
requested. However, please do distinguish the quoted part from your
comments. The first paragraph actually was written by John, but your
post lacks any hint of the author, and even worse displays the quote and
your text visually identical.
See the difference between your latest two posts and any other post in
this thread?
I blame Nabble for even making this possible. In a reply, the quoted
text must be visually distinctive. More reason to avoid Nabble.
> View this message in context: http://spamassassin.1065346.n5.nabble.com/Delays-with-Check-Bayes-tp111067p111118.html
> Sent from the SpamAssassin - Users mailing list archive at Nabble.com.
^^^^^^^^^^^^
Sic. This is a mailing list. And Nabble a third-party list archive
service and poor forum-style web frontend to the mailing list.
--
char *t="\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i<l;i++){ i%8? c<<=1:
(c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}
Re: Delays with Check_Bayes
Posted by redtailjason <ja...@redtailtechnology.com>.
On Wed, 20 Aug 2014, redtailjason wrote:
> We are seeing about 4000-7000 delayed messages per day. We do utilize a
> dedicated MySQL Server for the Bayes and all 8 scanners share it.
Are you open to the possibility of upgrading to 3.4.0 and using the Redis
backend for Bayes? (Just offering an alternative.)
We have been developing and upgrade plan to 3.4. Based on this, we are
prioritize this upgrade and will be expediting it. Thanks.
Jason
--
View this message in context: http://spamassassin.1065346.n5.nabble.com/Delays-with-Check-Bayes-tp111067p111118.html
Sent from the SpamAssassin - Users mailing list archive at Nabble.com.
Re: Delays with Check_Bayes
Posted by redtailjason <ja...@redtailtechnology.com>.
On 21/08/14 09:00, John Hardin wrote:
>
> Are you open to the possibility of upgrading to 3.4.0 and using the
> Redis backend for Bayes? (Just offering an alternative.)
>
We just last week moved over to 3.4.0 with a central Redis backend with
6 spamd servers spread over USA and Europe. Bit of a stretch in terms of
WAN latency but it seems to be working really well. I love doing a
"spamc -L spam" against one SA server and then immediately re-scanning
the same message by a different one and seeing the BAYES_99 light up :-)
So far, sooooo good!
Thanks for sharing this. We will take a look at that to see if it works for
us.
Regards,
Jason
--
View this message in context: http://spamassassin.1065346.n5.nabble.com/Delays-with-Check-Bayes-tp111067p111119.html
Sent from the SpamAssassin - Users mailing list archive at Nabble.com.
Re: Delays with Check_Bayes
Posted by Jason Haar <Ja...@trimble.com>.
On 21/08/14 09:00, John Hardin wrote:
>
> Are you open to the possibility of upgrading to 3.4.0 and using the
> Redis backend for Bayes? (Just offering an alternative.)
>
We just last week moved over to 3.4.0 with a central Redis backend with
6 spamd servers spread over USA and Europe. Bit of a stretch in terms of
WAN latency but it seems to be working really well. I love doing a
"spamc -L spam" against one SA server and then immediately re-scanning
the same message by a different one and seeing the BAYES_99 light up :-)
So far, sooooo good!
--
Cheers
Jason Haar
Corporate Information Security Manager, Trimble Navigation Ltd.
Phone: +1 408 481 8171
PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1
Re: Delays with Check_Bayes
Posted by John Hardin <jh...@impsec.org>.
On Wed, 20 Aug 2014, redtailjason wrote:
> We are seeing about 4000-7000 delayed messages per day. We do utilize a
> dedicated MySQL Server for the Bayes and all 8 scanners share it.
Are you open to the possibility of upgrading to 3.4.0 and using the Redis
backend for Bayes? (Just offering an alternative.)
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
If Microsoft made hammers, everyone would whine about how poorly
screws were designed and about how they are hard to hammer in, and
wonder why it takes so long to paint a wall using the hammer.
-----------------------------------------------------------------------
4 days until the 1935th anniversary of the destruction of Pompeii
Re: Delays with Check_Bayes
Posted by redtailjason <ja...@redtailtechnology.com>.
We are seeing about 4000-7000 delayed messages per day. We do utilize a
dedicated MySQL Server for the Bayes and all 8 scanners share it. Please let
me know if this does not fully clarify our setup for you.
Has anyone heard of a configuration where the transactions are written to a
file (ie text file) and then inserted into the database every few minutes?
MySQL seems to process imports faster than line by line transactions.
--
View this message in context: http://spamassassin.1065346.n5.nabble.com/Delays-with-Check-Bayes-tp111067p111092.html
Sent from the SpamAssassin - Users mailing list archive at Nabble.com.
Re: Delays with Check_Bayes
Posted by Karsten Bräckelmann <gu...@rudersport.de>.
On Wed, 2014-08-20 at 06:15 -0700, redtailjason wrote:
> Hello and good morning. We are running into some delays that we are trying to
> pin down a root cause for.
>
> Below are some examples. Within the examples, you can see that the
> check_bayes: scan is consuming most of the timing. Does anyone have any
> suggests on what to look at? We use 3.3.2. We have eight scanners setup to
> handle the scanning with 5GB RAM and 4 CPUs each. Volume is 250K - 500K per
> day.
That volume means throughput of about 350 messages per minute, 5.8 per
second. Sounds reasonable for 8 dedicated scanners.
Your samples are showing overall timings between about 90 seconds and
more than 2 minutes. Which means processing commonly takes less time,
and these are some extreme cases -- unless you really do have 50-100
busy processes per machine.
How many such long-running processes do you see, how frequent are they?
Also, you mentioned you are using the MySQL backend for Bayes. You did
not add any further detail, though.
Do you have dedicated MySQL servers for Bayes? Or does each scanner
machine run a local MySQL server? Do they share / sync databases
somehow?
Please elaborate on your environment, in particular everything
concerning Bayes.
--
char *t="\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i<l;i++){ i%8? c<<=1:
(c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}