You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by hg user <me...@gmail.com> on 2019/06/28 08:49:52 UTC
amavisd 100% cpu load - 470 queued messages...
I'm not able to lower cpu usage of amavisd.
4 cpus are used 100% and messages queue up to 15 minutes before being
processed.
mailq reports up to 470 queued messages... and this is bad, really bad.
The most part of SA work is spent here:
tests_pri_0: 7371 (93.7%)
and I know that priority 0 includes almost everything....
Other than disabling some rules (which one?!!?!) and adding some cpus, what
can I really do?
Re: amavisd 100% cpu load - 470 queued messages...
Posted by Henrik K <he...@hege.li>.
On Fri, Jun 28, 2019 at 12:03:23PM +0200, hg user wrote:
>
> A message I was able to isolate is 94kb, has text and html part and processing
> via command line takes about 6 seconds, I received 500+ of them. Due to
> unsubscribe urls containing the mail address, all messages have different
> hashes
If it happens to be large pathological body for some regexes, upcoming 3.4.3
might help here with it's body_part_scan_size option (truncates body length
to 50k by default, fine to drop to 10-20k on slow server too).
3.4.3-rc3 is available here if you or someone wants to try:
http://talon2.pccc.com/~kmcgrail/devel/
To identify slow rules, download
http://svn.apache.org/repos/asf/spamassassin/trunk/masses/plugins/HitFreqsRuleTiming.pm
And run
spamassassin --cf 'loadplugin HitFreqsRuleTiming /path/to/HitFreqsRuleTiming.pm' -L messagefile
See resulting timing.log file
Re: amavisd 100% cpu load - 470 queued messages...
Posted by hg user <me...@gmail.com>.
In this moment I have more than 400 delivery of a 178kb text/html
message... no attachment...
For specific senders I may:
- apply a very restrictive throttling
- skip spamassassin check
On Fri, Jun 28, 2019 at 3:18 PM Matus UHLAR - fantomas <uh...@fantomas.sk>
wrote:
> >> On 28.06.19 12:03, hg user wrote:
> >> >I did investigate a bit and was able to link spikes in emails received
> on
> >> >the external MTAs to spikes in messages queued in amavisd.
> >> >
> >> >As soon as I receive a mailing list/newsletter kind of email addressed
> to
> >> >several hundreds recipients and the message size is more than 70kb, the
> >> >messages starts to queue up.
> >> >
> >> >A message I was able to isolate is 94kb, has text and html part and
> >> >processing via command line takes about 6 seconds, I received 500+ of
> them.
> >> >Due to unsubscribe urls containing the mail address, all messages have
> >> >different hashes
>
> >On Fri, Jun 28, 2019 at 01:23:38PM +0200, Matus UHLAR - fantomas wrote:
> >> 6 seconds message scanning is quite fast. I have set up
> >> razor,pyzor,dcc,dnsbl timeouts to 20 seconds to improve
> >> acurracy (maybe marginally, but I don't care).
> >>
> >> If you receive that much mail often, you need faster computer.
> >> Maybe sa-compile would help you a bit, but only with CPU usage, network
> >> checks need time (and they are effective so don't mess them up).
>
> On 28.06.19 15:09, Henrik K wrote:
> >If the bottleneck is network checks, one should just increase parallel
> >processes. But OP probably has some other problems if regex are taking so
> >long (suggested by the 100% CPU usage).
>
> processing 370 of 70k emails within a few seconds of course results in 100%
> cpu usage for those few seconds :-)
>
> >Biggest improvement comes from razor/pyzor/dcc being asynchronous in
> trunk,
> >I suggest trying it if possible. :-)
>
> great news.
> --
> Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
> Warning: I wish NOT to receive e-mail advertising to this address.
> Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
> Eagles may soar, but weasels don't get sucked into jet engines.
>
Re: amavisd 100% cpu load - 470 queued messages...
Posted by Matus UHLAR - fantomas <uh...@fantomas.sk>.
>> On 28.06.19 12:03, hg user wrote:
>> >I did investigate a bit and was able to link spikes in emails received on
>> >the external MTAs to spikes in messages queued in amavisd.
>> >
>> >As soon as I receive a mailing list/newsletter kind of email addressed to
>> >several hundreds recipients and the message size is more than 70kb, the
>> >messages starts to queue up.
>> >
>> >A message I was able to isolate is 94kb, has text and html part and
>> >processing via command line takes about 6 seconds, I received 500+ of them.
>> >Due to unsubscribe urls containing the mail address, all messages have
>> >different hashes
>On Fri, Jun 28, 2019 at 01:23:38PM +0200, Matus UHLAR - fantomas wrote:
>> 6 seconds message scanning is quite fast. I have set up
>> razor,pyzor,dcc,dnsbl timeouts to 20 seconds to improve
>> acurracy (maybe marginally, but I don't care).
>>
>> If you receive that much mail often, you need faster computer.
>> Maybe sa-compile would help you a bit, but only with CPU usage, network
>> checks need time (and they are effective so don't mess them up).
On 28.06.19 15:09, Henrik K wrote:
>If the bottleneck is network checks, one should just increase parallel
>processes. But OP probably has some other problems if regex are taking so
>long (suggested by the 100% CPU usage).
processing 370 of 70k emails within a few seconds of course results in 100%
cpu usage for those few seconds :-)
>Biggest improvement comes from razor/pyzor/dcc being asynchronous in trunk,
>I suggest trying it if possible. :-)
great news.
--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Eagles may soar, but weasels don't get sucked into jet engines.
Re: amavisd 100% cpu load - 470 queued messages...
Posted by Henrik K <he...@hege.li>.
On Fri, Jun 28, 2019 at 01:23:38PM +0200, Matus UHLAR - fantomas wrote:
> On 28.06.19 12:03, hg user wrote:
> >I did investigate a bit and was able to link spikes in emails received on
> >the external MTAs to spikes in messages queued in amavisd.
> >
> >As soon as I receive a mailing list/newsletter kind of email addressed to
> >several hundreds recipients and the message size is more than 70kb, the
> >messages starts to queue up.
> >
> >A message I was able to isolate is 94kb, has text and html part and
> >processing via command line takes about 6 seconds, I received 500+ of them.
> >Due to unsubscribe urls containing the mail address, all messages have
> >different hashes
>
> 6 seconds message scanning is quite fast. I have set up
> razor,pyzor,dcc,dnsbl timeouts to 20 seconds to improve
> acurracy (maybe marginally, but I don't care).
>
> If you receive that much mail often, you need faster computer.
> Maybe sa-compile would help you a bit, but only with CPU usage, network
> checks need time (and they are effective so don't mess them up).
If the bottleneck is network checks, one should just increase parallel
processes. But OP probably has some other problems if regex are taking so
long (suggested by the 100% CPU usage).
Here's some figures for trunk/4.0.0 with razor,pyzor,dcc,ixhash and bunch of
other stuff.
# grep TIMING-SA mail.log mail.log.1 | pcregrep -o '\d+ ms' | awk '{print $1}' | histogram
Count: 4375
Range: 7.000 - 8551.000; Mean: 1720.447; Median: 1252.000; Stddev: 1632.929
Percentiles: 90th: 4422.000; 95th: 4589.000; 99th: 8101.000
7.000 - 15.069: 36 ##
15.069 - 31.276: 184 #########
31.276 - 63.831: 426 ####################
63.831 - 129.221: 335 ################
129.221 - 260.565: 113 #####
260.565 - 524.384: 14 #
524.384 - 1054.296: 545 #########################
1054.296 - 2118.689: 1138 #####################################################
2118.689 - 4256.650: 945 ############################################
4256.650 - 8551.000: 639 ##############################
<500ms are shortcircuits. But mostly 1-4 sec for normal stuff.
Biggest improvement comes from razor/pyzor/dcc being asynchronous in trunk,
I suggest trying it if possible. :-)
Re: amavisd 100% cpu load - 470 queued messages...
Posted by Matus UHLAR - fantomas <uh...@fantomas.sk>.
On 28.06.19 12:03, hg user wrote:
>I did investigate a bit and was able to link spikes in emails received on
>the external MTAs to spikes in messages queued in amavisd.
>
>As soon as I receive a mailing list/newsletter kind of email addressed to
>several hundreds recipients and the message size is more than 70kb, the
>messages starts to queue up.
>
>A message I was able to isolate is 94kb, has text and html part and
>processing via command line takes about 6 seconds, I received 500+ of them.
>Due to unsubscribe urls containing the mail address, all messages have
>different hashes
6 seconds message scanning is quite fast.
I have set up razor,pyzor,dcc,dnsbl timeouts to 20 seconds to improve
acurracy (maybe marginally, but I don't care).
If you receive that much mail often, you need faster computer.
Maybe sa-compile would help you a bit, but only with CPU usage, network
checks need time (and they are effective so don't mess them up).
--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Atheism is a non-prophet organization.
Re: amavisd 100% cpu load - 470 queued messages...
Posted by hg user <me...@gmail.com>.
I did investigate a bit and was able to link spikes in emails received on
the external MTAs to spikes in messages queued in amavisd.
As soon as I receive a mailing list/newsletter kind of email addressed to
several hundreds recipients and the message size is more than 70kb, the
messages starts to queue up.
A message I was able to isolate is 94kb, has text and html part and
processing via command line takes about 6 seconds, I received 500+ of them.
Due to unsubscribe urls containing the mail address, all messages have
different hashes
On Fri, Jun 28, 2019 at 11:38 AM Matus UHLAR - fantomas <uh...@fantomas.sk>
wrote:
> >> On Fri, Jun 28, 2019 at 10:49 AM hg user <me...@gmail.com>
> wrote:
> >>> I'm not able to lower cpu usage of amavisd.
> >>> 4 cpus are used 100% and messages queue up to 15 minutes before being
> >>> processed.
> >>>
> >>> mailq reports up to 470 queued messages... and this is bad, really bad.
> >>>
> >>> The most part of SA work is spent here:
> >>> tests_pri_0: 7371 (93.7%)
> >>> and I know that priority 0 includes almost everything....
> >>>
> >>> Other than disabling some rules (which one?!!?!) and adding some cpus,
> >>> what can I really do?
>
> >> Messages reported by mailq decreased to about 370 and then, in a few
> >> seconds, to 0... from 370 to 0 in a few seconds...
>
> On 28.06.19 10:03, Dominic Raferd wrote:
> >As a workaround you could set (in amavis conf file) something like:
> >
> >$child_timeout = 20;
> >
> >which restricts any child process to (roughly) 20 seconds after which
> >amavis will abort it
>
> I don't think that's a good idea. That would highly decreate accuracy.
>
> --
> Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
> Warning: I wish NOT to receive e-mail advertising to this address.
> Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
> Nothing is fool-proof to a talented fool.
>
Re: amavisd 100% cpu load - 470 queued messages...
Posted by Matus UHLAR - fantomas <uh...@fantomas.sk>.
>> On Fri, Jun 28, 2019 at 10:49 AM hg user <me...@gmail.com> wrote:
>>> I'm not able to lower cpu usage of amavisd.
>>> 4 cpus are used 100% and messages queue up to 15 minutes before being
>>> processed.
>>>
>>> mailq reports up to 470 queued messages... and this is bad, really bad.
>>>
>>> The most part of SA work is spent here:
>>> tests_pri_0: 7371 (93.7%)
>>> and I know that priority 0 includes almost everything....
>>>
>>> Other than disabling some rules (which one?!!?!) and adding some cpus,
>>> what can I really do?
>> Messages reported by mailq decreased to about 370 and then, in a few
>> seconds, to 0... from 370 to 0 in a few seconds...
On 28.06.19 10:03, Dominic Raferd wrote:
>As a workaround you could set (in amavis conf file) something like:
>
>$child_timeout = 20;
>
>which restricts any child process to (roughly) 20 seconds after which
>amavis will abort it
I don't think that's a good idea. That would highly decreate accuracy.
--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Nothing is fool-proof to a talented fool.
Re: amavisd 100% cpu load - 470 queued messages...
Posted by Dominic Raferd <do...@timedicer.co.uk>.
On Fri, 28 Jun 2019 at 09:56, hg user <me...@gmail.com> wrote:
> Messages reported by mailq decreased to about 370 and then, in a few
> seconds, to 0... from 370 to 0 in a few seconds...
>
>
>
> On Fri, Jun 28, 2019 at 10:49 AM hg user <me...@gmail.com> wrote:
>
>> I'm not able to lower cpu usage of amavisd.
>> 4 cpus are used 100% and messages queue up to 15 minutes before being
>> processed.
>>
>> mailq reports up to 470 queued messages... and this is bad, really bad.
>>
>> The most part of SA work is spent here:
>> tests_pri_0: 7371 (93.7%)
>> and I know that priority 0 includes almost everything....
>>
>> Other than disabling some rules (which one?!!?!) and adding some cpus,
>> what can I really do?
>>
>
As a workaround you could set (in amavis conf file) something like:
$child_timeout = 20;
which restricts any child process to (roughly) 20 seconds after which
amavis will abort it
Re: amavisd 100% cpu load - 470 queued messages...
Posted by hg user <me...@gmail.com>.
Messages reported by mailq decreased to about 370 and then, in a few
seconds, to 0... from 370 to 0 in a few seconds...
On Fri, Jun 28, 2019 at 10:49 AM hg user <me...@gmail.com> wrote:
> I'm not able to lower cpu usage of amavisd.
> 4 cpus are used 100% and messages queue up to 15 minutes before being
> processed.
>
> mailq reports up to 470 queued messages... and this is bad, really bad.
>
> The most part of SA work is spent here:
> tests_pri_0: 7371 (93.7%)
> and I know that priority 0 includes almost everything....
>
> Other than disabling some rules (which one?!!?!) and adding some cpus,
> what can I really do?
>