You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Bill Moseley <mo...@hank.org> on 2005/01/10 19:59:27 UTC
spamd eating memory
I've got a low traffic mail server running Debian Woody with backports
for SA and Exim:
Exim: 4.34-7.amwoody.1 0
SA: 3.0.2-1
Perl: 5.6.1
Again, low traffic. I rotate daily, so:
$ wc -l /var/log/exim4/mainlog.1
2467 /var/log/exim4/mainlog.1
I just upgraded a few days ago to 3.0 from 2.6.4. The machine is a P4
with 1/2G RAM.
I'm seeing lots of these in my logs:
kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
VM: killing process spamd
You can see here how I run spamd and how there's one child that is a
bit of a memory hog:
ps -aux | grep spamd
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 16610 0.0 0.1 92440 732 ? S Jan05 0:04 /usr/sbin/spamd --max-children 5 -d --pidfile=/var/run/spamd.pid
root 25927 1.7 26.6 439648 137676 ? S Jan09 23:23 spamd child
root 5931 0.1 13.2 98728 68588 ? S 05:46 0:17 spamd child
root 8734 0.0 13.2 98280 68456 ? S 08:41 0:05 spamd child
root 8755 0.0 13.1 97652 67852 ? S 08:45 0:05 spamd child
root 8785 0.1 13.6 100148 70552 ? S 08:49 0:06 spamd child
root 10432 0.0 0.0 1332 428 pts/6 R 10:33 0:00 grep spamd
I run spamd via exiscan-acl version of Exim
warn message = X-Spam-Score: $spam_score ($spam_bar)
spam = nobody:true
warn message = X-Spam-Report: $spam_report
spam = nobody:true
deny message = This message scored $spam_score spam points.
spam = nobody:true
condition = ${if >{$spam_score_int}{100}{1}{0}}
I suspect --max-conn-per-child is a good protection. Any
recommendation on what to set it to?
Is this a known issue?
--
Bill Moseley
moseley@hank.org
Re: spamd eating memory
Posted by Ryan Castellucci <ry...@gmail.com>.
I've been seeing the same thing on RedHat 9, with spamassassin 3.0.2.
Anyone have a fix?
On Mon, 10 Jan 2005 13:24:27 -0800, Bill Moseley <mo...@hank.org> wrote:
> It's interesting how one process (of all started at the same time) is
> using so much memory.
>
> # ps aux | egrep '(spamd|USER)'
> USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
> root 10437 0.0 3.4 92440 17896 ? S 10:33 0:04 /usr/sbin/spamd --max-children 5 -d --pidfile=/var/run/spamd.pid
> root 10442 0.1 13.3 97776 69104 ? S 10:33 0:13 spamd child
> root 10443 2.3 48.0 433904 247716 ? S 10:33 3:54 spamd child
> root 10444 0.1 13.2 97228 68288 ? S 10:33 0:12 spamd child
> root 10445 0.1 13.4 98488 69168 ? S 10:33 0:12 spamd child
> root 10446 0.1 14.3 102836 74116 ? S 10:33 0:15 spamd child
> root 13067 0.0 0.0 1328 428 pts/6 R 13:20 0:00 egrep (spamd|USER)
>
>
> --
> Bill Moseley
> moseley@hank.org
>
>
Re: spamd eating memory
Posted by Bill Moseley <mo...@hank.org>.
It's interesting how one process (of all started at the same time) is
using so much memory.
# ps aux | egrep '(spamd|USER)'
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 10437 0.0 3.4 92440 17896 ? S 10:33 0:04 /usr/sbin/spamd --max-children 5 -d --pidfile=/var/run/spamd.pid
root 10442 0.1 13.3 97776 69104 ? S 10:33 0:13 spamd child
root 10443 2.3 48.0 433904 247716 ? S 10:33 3:54 spamd child
root 10444 0.1 13.2 97228 68288 ? S 10:33 0:12 spamd child
root 10445 0.1 13.4 98488 69168 ? S 10:33 0:12 spamd child
root 10446 0.1 14.3 102836 74116 ? S 10:33 0:15 spamd child
root 13067 0.0 0.0 1328 428 pts/6 R 13:20 0:00 egrep (spamd|USER)
--
Bill Moseley
moseley@hank.org
Re: spamd eating memory
Posted by Martin Hepworth <ma...@solid-state-logic.com>.
Bill
wouldn't like to comment on the changes, I just see people see get
positive results from it (ie lowest memory usage).
I don't use spamd to invoke spamassassin myself so haven't suffered from
the problem. (I use MailScanner)
--
Martin Hepworth
Snr Systems Administrator
Solid State Logic
Tel: +44 (0)1865 842300
Bill Moseley wrote:
> On Wed, Jan 12, 2005 at 09:43:33AM +0000, Martin Hepworth wrote:
>
>>There's a bugilla reference to this and a patch for the patch there as well
>
>
>>http://bugzilla.spamassassin.org/show_bug.cgi?id=3983
>
>
> That patch is about a change in how the spamd processes are used,
> right? Instead of using all --max-children (as Linux decides) to
> using a smaller pool of them and trying to use ones that are not
> swapped to disk.
>
> Maybe I was not letting SA run long enough, but in my case it seems
> like only one spamd process is growing large (see my other posts for ps
> output). Most were 100K but 1 was almost 5 times larger.
>
>
>
**********************************************************************
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.
This footnote confirms that this email message has been swept
for the presence of computer viruses and is believed to be clean.
**********************************************************************
Re: spamd eating memory
Posted by Bill Moseley <mo...@hank.org>.
On Wed, Jan 12, 2005 at 09:43:33AM +0000, Martin Hepworth wrote:
> There's a bugilla reference to this and a patch for the patch there as well
> http://bugzilla.spamassassin.org/show_bug.cgi?id=3983
That patch is about a change in how the spamd processes are used,
right? Instead of using all --max-children (as Linux decides) to
using a smaller pool of them and trying to use ones that are not
swapped to disk.
Maybe I was not letting SA run long enough, but in my case it seems
like only one spamd process is growing large (see my other posts for ps
output). Most were 100K but 1 was almost 5 times larger.
--
Bill Moseley
moseley@hank.org
Re: spamd eating memory
Posted by Martin Hepworth <ma...@solid-state-logic.com>.
Bill
gmame's search was screwed yesterday. will try agian today and bookmark
the thing ;-)
ok the search term on gmame you need is "low memory"
There's a bugilla reference to this and a patch for the patch there as well
http://bugzilla.spamassassin.org/show_bug.cgi?id=3983
--
Martin Hepworth
Snr Systems Administrator
Solid State Logic
Tel: +44 (0)1865 842300
Bill Moseley wrote:
> On Tue, Jan 11, 2005 at 09:14:03AM +0000, Martin Hepworth wrote:
>
>>Bill
>>
>>
>>Alot of people seem to be noticed this, there is a small memory patch
>>floating around the email list archives you could try.
>
>
> Do you remember any good keywords subject or author to help in
> locating the patch? I'm not having much luck searching.
>
> By the way, I set --max-conn-per-child=20 (yes, very low) and that
> seems to have solved the problem with a given spamd process eating
> memory.
>
> But I'm still seeing a lot of:
>
> Jan 11 10:03:57 mardy spamd[13158]: server hit by SIGCHLD
> Jan 11 10:03:57 mardy spamd[13158]: handled cleanup of child pid 27032
> Jan 11 10:03:57 mardy spamd[13158]: server successfully spawned child process, pid 30366
> Jan 11 10:03:57 mardy spamd[13158]: server hit by SIGCHLD
> Jan 11 10:03:57 mardy spamd[13158]: handled cleanup of child pid 27032
> Jan 11 10:03:57 mardy spamd[13158]: server successfully spawned child process, pid 30366
> Jan 11 10:03:57 mardy spamd[13158]: server hit by SIGCHLD
> [...]
>
> Is that a result of the child process being killed by
> --max-conn-perl-child?
>
>
> Thanks,
>
**********************************************************************
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.
This footnote confirms that this email message has been swept
for the presence of computer viruses and is believed to be clean.
**********************************************************************
Re: spamd eating memory
Posted by Michael Parker <pa...@pobox.com>.
On Wed, Jan 12, 2005 at 07:19:38AM -0800, Bill Moseley wrote:
>
> Since only one child process is so large and that they are all stared
> at the same time (e.g. processed the same number of requests) I would
> think its memory usage would be related to a specific message that
> spamd process handled.
>
Do you limit the size of the messages you send to spamd?
Michael
Re: spamd eating memory
Posted by Bill Moseley <mo...@hank.org>.
On Tue, Jan 11, 2005 at 09:57:02PM -0800, Loren Wilton wrote:
> BTW, this probably means that you have a really big whitelist or bayes db,
> and possibly expire isn't working as you think it is. Might be worth
> checking on that.
I'm not using bayes (AFAIK ;), as I'm doing site-wide filtering with
Exiscan and I don't have a bayes_path setup in my local.cf.
My read of the docs I see auot-whitelisting is enabled by default now,
but all my spam filtering is done at smtp data time (via exiscan) so
I'm not sure where the whitelist would be stored. SA isn't run for
each user so their .spamassassin dir is empty (other than the default
user_prefs). lsof for spamd didn't give me any hints, either.
Since only one child process is so large and that they are all stared
at the same time (e.g. processed the same number of requests) I would
think its memory usage would be related to a specific message that
spamd process handled.
$ ps auxf | egrep '(spamd|USER)'
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 13158 0.0 3.4 92440 17548 ? S Jan10 0:04 /usr/sbin/spamd --max-children 5 --max-conn-per-child=20 -d --pidfile=/var/run/spamd.pid
root 11081 0.1 13.7 99320 71024 ? S 03:06 0:16 \_ spamd child
root 11093 0.1 15.3 107640 79100 ? S 03:06 0:19 \_ spamd child
root 11137 0.0 13.3 97580 68736 ? S 03:14 0:11 \_ spamd child
root 11172 0.4 21.6 181620 111888 ? S 03:22 1:06 \_ spamd child
root 11197 0.1 13.8 100368 71384 ? S 03:23 0:15 \_ spamd child
Probably unrelated, but what does "autolearn=failed" mean?
Jan 12 06:38:28 mardy spamd[11172]: result: . 0 - scantime=0.0,size=1955,mid=<95...@mail.nih.gov>,autolearn=failed
--
Bill Moseley
moseley@hank.org
Re: spamd eating memory
Posted by Loren Wilton <lw...@earthlink.net>.
> By the way, I set --max-conn-per-child=20 (yes, very low) and that
> seems to have solved the problem with a given spamd process eating
> memory.
>
> But I'm still seeing a lot of:
>
> Jan 11 10:03:57 mardy spamd[13158]: server hit by SIGCHLD
> Is that a result of the child process being killed by
> --max-conn-perl-child?
Its a result of reaching the max number of connections per child, and is
perfectly normal, despite looking like an error. I think I may have opened
a bug to make this look less like an error, but I don't recall for sure if I
did or not.
You would see this (eventually) even without setting max conn, it would just
take longer.
Loren
BTW, this probably means that you have a really big whitelist or bayes db,
and possibly expire isn't working as you think it is. Might be worth
checking on that.
Re: spamd eating memory
Posted by Bill Moseley <mo...@hank.org>.
On Tue, Jan 11, 2005 at 09:14:03AM +0000, Martin Hepworth wrote:
> Bill
>
>
> Alot of people seem to be noticed this, there is a small memory patch
> floating around the email list archives you could try.
Do you remember any good keywords subject or author to help in
locating the patch? I'm not having much luck searching.
By the way, I set --max-conn-per-child=20 (yes, very low) and that
seems to have solved the problem with a given spamd process eating
memory.
But I'm still seeing a lot of:
Jan 11 10:03:57 mardy spamd[13158]: server hit by SIGCHLD
Jan 11 10:03:57 mardy spamd[13158]: handled cleanup of child pid 27032
Jan 11 10:03:57 mardy spamd[13158]: server successfully spawned child process, pid 30366
Jan 11 10:03:57 mardy spamd[13158]: server hit by SIGCHLD
Jan 11 10:03:57 mardy spamd[13158]: handled cleanup of child pid 27032
Jan 11 10:03:57 mardy spamd[13158]: server successfully spawned child process, pid 30366
Jan 11 10:03:57 mardy spamd[13158]: server hit by SIGCHLD
[...]
Is that a result of the child process being killed by
--max-conn-perl-child?
Thanks,
--
Bill Moseley
moseley@hank.org
Re: spamd eating memory
Posted by Martin Hepworth <ma...@solid-state-logic.com>.
Bill
Alot of people seem to be noticed this, there is a small memory patch
floating around the email list archives you could try.
It missed the 3.02 cut off, but apparently works fine with this version.
--
Martin Hepworth
Snr Systems Administrator
Solid State Logic
Tel: +44 (0)1865 842300
Bill Moseley wrote:
> I've got a low traffic mail server running Debian Woody with backports
> for SA and Exim:
>
> Exim: 4.34-7.amwoody.1 0
> SA: 3.0.2-1
> Perl: 5.6.1
>
> Again, low traffic. I rotate daily, so:
>
> $ wc -l /var/log/exim4/mainlog.1
> 2467 /var/log/exim4/mainlog.1
>
>
> I just upgraded a few days ago to 3.0 from 2.6.4. The machine is a P4
> with 1/2G RAM.
>
> I'm seeing lots of these in my logs:
>
> kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
> VM: killing process spamd
>
> You can see here how I run spamd and how there's one child that is a
> bit of a memory hog:
>
> ps -aux | grep spamd
> USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
> root 16610 0.0 0.1 92440 732 ? S Jan05 0:04 /usr/sbin/spamd --max-children 5 -d --pidfile=/var/run/spamd.pid
> root 25927 1.7 26.6 439648 137676 ? S Jan09 23:23 spamd child
> root 5931 0.1 13.2 98728 68588 ? S 05:46 0:17 spamd child
> root 8734 0.0 13.2 98280 68456 ? S 08:41 0:05 spamd child
> root 8755 0.0 13.1 97652 67852 ? S 08:45 0:05 spamd child
> root 8785 0.1 13.6 100148 70552 ? S 08:49 0:06 spamd child
> root 10432 0.0 0.0 1332 428 pts/6 R 10:33 0:00 grep spamd
>
> I run spamd via exiscan-acl version of Exim
>
> warn message = X-Spam-Score: $spam_score ($spam_bar)
> spam = nobody:true
>
> warn message = X-Spam-Report: $spam_report
> spam = nobody:true
>
> deny message = This message scored $spam_score spam points.
> spam = nobody:true
> condition = ${if >{$spam_score_int}{100}{1}{0}}
>
> I suspect --max-conn-per-child is a good protection. Any
> recommendation on what to set it to?
>
> Is this a known issue?
>
>
**********************************************************************
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.
This footnote confirms that this email message has been swept
for the presence of computer viruses and is believed to be clean.
**********************************************************************