You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Len Conrad <LC...@Go2France.com> on 2010/09/05 19:33:06 UTC
Re: scantime=249.2; scantime=175.0; scantime=190.9;
scantime=68.9
>Mem: 772880k total, 685316k used, 87564k free, 31344k buffers
>Swap: 1076312k total, 249032k used, 827280k free, 156328k cached
250MB swapped, for less than 1 GB RAM, used is disastrous for an MTA.
Increase RAM to 2GB, or until swap is always "0k used"
Len
Re: scantime=249.2; scantime=175.0; scantime=190.9; scantime=68.9
Posted by RW <rw...@googlemail.com>.
On Mon, 06 Sep 2010 17:03:02 +0200
Karsten Bräckelmann <gu...@rudersport.de> wrote:
> Since you mentioned procmail, your spamc calling recipe is *with*
> locking, right? Limiting concurrent SA processes pretty much to one as
> far as filtering is concerned.
And start spamd with --max-children=1. That not only frees up
processes, but with --max-conn-per-child=3 it avoids frequently
replacing the "hot child" with a paged-out spare.
Re: scantime=249.2; scantime=175.0; scantime=190.9; scantime=68.9
Posted by Chris <cp...@embarqmail.com>.
On Tue, 2010-09-07 at 00:54 +0200, Karsten Bräckelmann wrote:
> On Mon, 2010-09-06 at 17:32 -0500, Chris wrote:
> > On Mon, 2010-09-06 at 17:03 +0200, Karsten Bräckelmann wrote:
>
> > > Unless the limit of 50k results in quite some spam ending up unprocessed
> > > by SA, I doubt this will help.
> > >
> > > Dropping large-ish third-party rule sets, if any, is much more likely to
> > > make a noticeable difference. SA, as well as ClamAV. If memory serves me
> > > right, you are using ClamAV third-party signatures -- some of which have
> > > been reported to hog memory galore.
>
> > > Since you mentioned procmail, your spamc calling recipe is *with*
> > > locking, right? Limiting concurrent SA processes pretty much to one as
> > > far as filtering is concerned. (As Bernd and previously others in this
> > > thread have pointed out, limit the concurrency.)
> >
> > I believe I have this right Karsten
> >
> > :0 fw : $ASSASSINLOCK
>
> Indeed, you do. Well, if $ASSASSINLOCK evaluates to something, always
> evaluates to the same, and actually is writable. ;)
>
> > * < 150000
> > | /usr/local/bin/spamc -f
> >
> > I've trimmed my third party rule sets down to the sought rules and
> > disabled unnecessary rule sets I had setup in my local.cf
>
> Good one, particular if constrained with that "unnecessary". That part
> of my previous post strongly aimed at ClamAV third-party sigs, though.
> (Ah, well, and not loading them twice. But I believe we got that
> settled. ;)
>
>
I removed some of the third party sigs also Karsten, and memory usage
for clamd has dropped considerably:
28239 clamav 20 0 160m 132m 5304 S 0.0 17.6 0:27.65 clamd
Memory usage does look a bit better than it did when I started this
thread:
Mem: 772880k total, 744904k used, 27976k free, 34384k buffers
Swap: 1076312k total, 167580k used, 908732k free, 135420k cached
Still going into swap but until I upgrade memory that will have to be
sufficient. I can probably still remove some of the third party clam
sigs and have it do what I want.
Chris
--
Chris
KeyID 0xE372A7DA98E6705C
Re: scantime=249.2; scantime=175.0; scantime=190.9; scantime=68.9
Posted by Karsten Bräckelmann <gu...@rudersport.de>.
On Mon, 2010-09-06 at 17:32 -0500, Chris wrote:
> On Mon, 2010-09-06 at 17:03 +0200, Karsten Bräckelmann wrote:
> > Unless the limit of 50k results in quite some spam ending up unprocessed
> > by SA, I doubt this will help.
> >
> > Dropping large-ish third-party rule sets, if any, is much more likely to
> > make a noticeable difference. SA, as well as ClamAV. If memory serves me
> > right, you are using ClamAV third-party signatures -- some of which have
> > been reported to hog memory galore.
> > Since you mentioned procmail, your spamc calling recipe is *with*
> > locking, right? Limiting concurrent SA processes pretty much to one as
> > far as filtering is concerned. (As Bernd and previously others in this
> > thread have pointed out, limit the concurrency.)
>
> I believe I have this right Karsten
>
> :0 fw : $ASSASSINLOCK
Indeed, you do. Well, if $ASSASSINLOCK evaluates to something, always
evaluates to the same, and actually is writable. ;)
> * < 150000
> | /usr/local/bin/spamc -f
>
> I've trimmed my third party rule sets down to the sought rules and
> disabled unnecessary rule sets I had setup in my local.cf
Good one, particular if constrained with that "unnecessary". That part
of my previous post strongly aimed at ClamAV third-party sigs, though.
(Ah, well, and not loading them twice. But I believe we got that
settled. ;)
--
char *t="\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i<l;i++){ i%8? c<<=1:
(c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}
Re: scantime=249.2; scantime=175.0; scantime=190.9; scantime=68.9
Posted by Chris <cp...@embarqmail.com>.
On Mon, 2010-09-06 at 17:03 +0200, Karsten Bräckelmann wrote:
> On Sun, 2010-09-05 at 17:44 -0500, Chris wrote:
> > Thanks for the input John, I can accept 30 or 45 seconds of drive access
> > however when it comes to 300 I can't accept that. And you're absolutely
> > correct, the problem is my lack of memory I realize that now.
>
> > Just one user, me, though I already have procmail setup to not pass mail
> > destined for mailing list through SA or ClamAv. I had SA set to check
> > message size less than 250k in my procmailrc I've dropped it to 50k and
>
> Unless the limit of 50k results in quite some spam ending up unprocessed
> by SA, I doubt this will help.
>
> Dropping large-ish third-party rule sets, if any, is much more likely to
> make a noticeable difference. SA, as well as ClamAV. If memory serves me
> right, you are using ClamAV third-party signatures -- some of which have
> been reported to hog memory galore.
>
>
> > will see what happens. I could probably drop some of the extra rulesets
> > I run also which would probably cut down on access until I can upgrade
> > my memory. I'm going to consider this thread closed then as being solved
> > with "upgrade to more memory". Many thanks to all who replied and
> > offered suggestions. And to the SA Team, keep up the great work!
>
> Since you mentioned procmail, your spamc calling recipe is *with*
> locking, right? Limiting concurrent SA processes pretty much to one as
> far as filtering is concerned. (As Bernd and previously others in this
> thread have pointed out, limit the concurrency.)
>
>
I believe I have this right Karsten
:0 fw : $ASSASSINLOCK
* < 150000
| /usr/local/bin/spamc -f
I've trimmed my third party rule sets down to the sought rules and
disabled unnecessary rule sets I had setup in my local.cf
--
Chris
KeyID 0xE372A7DA98E6705C
Re: scantime=249.2; scantime=175.0; scantime=190.9; scantime=68.9
Posted by Karsten Bräckelmann <gu...@rudersport.de>.
On Sun, 2010-09-05 at 17:44 -0500, Chris wrote:
> Thanks for the input John, I can accept 30 or 45 seconds of drive access
> however when it comes to 300 I can't accept that. And you're absolutely
> correct, the problem is my lack of memory I realize that now.
> Just one user, me, though I already have procmail setup to not pass mail
> destined for mailing list through SA or ClamAv. I had SA set to check
> message size less than 250k in my procmailrc I've dropped it to 50k and
Unless the limit of 50k results in quite some spam ending up unprocessed
by SA, I doubt this will help.
Dropping large-ish third-party rule sets, if any, is much more likely to
make a noticeable difference. SA, as well as ClamAV. If memory serves me
right, you are using ClamAV third-party signatures -- some of which have
been reported to hog memory galore.
> will see what happens. I could probably drop some of the extra rulesets
> I run also which would probably cut down on access until I can upgrade
> my memory. I'm going to consider this thread closed then as being solved
> with "upgrade to more memory". Many thanks to all who replied and
> offered suggestions. And to the SA Team, keep up the great work!
Since you mentioned procmail, your spamc calling recipe is *with*
locking, right? Limiting concurrent SA processes pretty much to one as
far as filtering is concerned. (As Bernd and previously others in this
thread have pointed out, limit the concurrency.)
--
char *t="\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i<l;i++){ i%8? c<<=1:
(c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}
Re: scantime=249.2; scantime=175.0; scantime=190.9; scantime=68.9
Posted by Chris <cp...@embarqmail.com>.
On Mon, 2010-09-06 at 12:09 +0200, Bernd Petrovitsch wrote:
> On Son, 2010-09-05 at 17:44 -0500, Chris wrote:
> > On Sun, 2010-09-05 at 12:54 -0700, John Hardin wrote:
> > > On Sun, 5 Sep 2010, Chris wrote:
> > >
> > > > On Sun, 2010-09-05 at 12:33 -0500, Len Conrad wrote:
> > > >>> Mem: 772880k total, 685316k used, 87564k free, 31344k buffers
> > > >>> Swap: 1076312k total, 249032k used, 827280k free, 156328k cached
> > > >>
> > > >> 250MB swapped, for less than 1 GB RAM, used is disastrous for an MTA.
> > > >>
> > > >> Increase RAM to 2GB, or until swap is always "0k used"
> > > >
> > > > It's just a single user home system. True, I probably do need to
> > > > increase ram but I 'don't' think this has a bearing on this issue though
> > > > I may be wrong.
> > >
> > > Your system is swapping. That kills performance, pretty much across the
> > > board. Either buy more memory or accept the impact of an underprovisioned
> > > machine on the performance of mail delivery.
> > >
> > > Do you really need your mail to be delivered _that_ promptly? If
> > > interactive performance is acceptable then what does it matter if an email
> > > (delivered in the background) takes 30 seconds or 300 seconds to be
> > > stuffed into your inbox?
> >
> > Thanks for the input John, I can accept 30 or 45 seconds of drive access
> > however when it comes to 300 I can't accept that. And you're absolutely
>
> If it's really drive access, it smells like trashing (or you have an
> awfully and incredibly slow I/O subsystem).
> 5 minutes is almost zero in the email world. SMTP never was real-time
> anyway.
>
> > correct, the problem is my lack of memory I realize that now.
>
> How many of the various daemons (spamd, clamd, mimedefang, MTAs) do you
> run (even potentially) in parallel?
> And how much (unshared) RSS do they take?
> Just limit that to, say 2 or 3 of each, if only to avoid trashing.
>
> FWIW I had a similar situation (on much larger hardware. But running 80
> mimedefang processes in parallel doesn't make too much sense anyways -
> even with 4 CPUs).
>
> Bernd
One spamd process now (had two before), one clamd process (since
stopped) and postfix to process spam reports and local mail
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
18330 root 20 0 66504 56m 632 S 0.0 7.5 0:00.00 spamd
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 11589 0.3 7.5 67936 58676 ? S 16:21 0:05
spamdchild
postfix 6582 0.0 0.1 4024 1252 ? S 15:53 0:00 pickup -l
-t fifo
postfix 20985 0.0 0.0 4164 684 ? S Aug04 0:05 qmgr -l
-t fifo -
--
Chris
KeyID 0xE372A7DA98E6705C
Re: scantime=249.2; scantime=175.0; scantime=190.9; scantime=68.9
Posted by Bernd Petrovitsch <be...@petrovitsch.priv.at>.
On Son, 2010-09-05 at 17:44 -0500, Chris wrote:
> On Sun, 2010-09-05 at 12:54 -0700, John Hardin wrote:
> > On Sun, 5 Sep 2010, Chris wrote:
> >
> > > On Sun, 2010-09-05 at 12:33 -0500, Len Conrad wrote:
> > >>> Mem: 772880k total, 685316k used, 87564k free, 31344k buffers
> > >>> Swap: 1076312k total, 249032k used, 827280k free, 156328k cached
> > >>
> > >> 250MB swapped, for less than 1 GB RAM, used is disastrous for an MTA.
> > >>
> > >> Increase RAM to 2GB, or until swap is always "0k used"
> > >
> > > It's just a single user home system. True, I probably do need to
> > > increase ram but I 'don't' think this has a bearing on this issue though
> > > I may be wrong.
> >
> > Your system is swapping. That kills performance, pretty much across the
> > board. Either buy more memory or accept the impact of an underprovisioned
> > machine on the performance of mail delivery.
> >
> > Do you really need your mail to be delivered _that_ promptly? If
> > interactive performance is acceptable then what does it matter if an email
> > (delivered in the background) takes 30 seconds or 300 seconds to be
> > stuffed into your inbox?
>
> Thanks for the input John, I can accept 30 or 45 seconds of drive access
> however when it comes to 300 I can't accept that. And you're absolutely
If it's really drive access, it smells like trashing (or you have an
awfully and incredibly slow I/O subsystem).
5 minutes is almost zero in the email world. SMTP never was real-time
anyway.
> correct, the problem is my lack of memory I realize that now.
How many of the various daemons (spamd, clamd, mimedefang, MTAs) do you
run (even potentially) in parallel?
And how much (unshared) RSS do they take?
Just limit that to, say 2 or 3 of each, if only to avoid trashing.
FWIW I had a similar situation (on much larger hardware. But running 80
mimedefang processes in parallel doesn't make too much sense anyways -
even with 4 CPUs).
Bernd
--
Bernd Petrovitsch Email : bernd@petrovitsch.priv.at
LUGA : http://www.luga.at
Re: scantime=249.2; scantime=175.0; scantime=190.9; scantime=68.9
Posted by Chris <cp...@embarqmail.com>.
On Sun, 2010-09-05 at 12:54 -0700, John Hardin wrote:
> On Sun, 5 Sep 2010, Chris wrote:
>
> > On Sun, 2010-09-05 at 12:33 -0500, Len Conrad wrote:
> >>> Mem: 772880k total, 685316k used, 87564k free, 31344k buffers
> >>> Swap: 1076312k total, 249032k used, 827280k free, 156328k cached
> >>
> >> 250MB swapped, for less than 1 GB RAM, used is disastrous for an MTA.
> >>
> >> Increase RAM to 2GB, or until swap is always "0k used"
> >
> > It's just a single user home system. True, I probably do need to
> > increase ram but I 'don't' think this has a bearing on this issue though
> > I may be wrong.
>
> Your system is swapping. That kills performance, pretty much across the
> board. Either buy more memory or accept the impact of an underprovisioned
> machine on the performance of mail delivery.
>
> Do you really need your mail to be delivered _that_ promptly? If
> interactive performance is acceptable then what does it matter if an email
> (delivered in the background) takes 30 seconds or 300 seconds to be
> stuffed into your inbox?
Thanks for the input John, I can accept 30 or 45 seconds of drive access
however when it comes to 300 I can't accept that. And you're absolutely
correct, the problem is my lack of memory I realize that now.
> How many email users is this supporting? You might want to consider some
> glue-level performance hacks - for example, if your box is supporting
> email for your family members, and all delivery for that is local, then
> you could configure your MTA to not pass locally-originated-and-destined
> messages through clamav or SA at all.
>
Just one user, me, though I already have procmail setup to not pass mail
destined for mailing list through SA or ClamAv. I had SA set to check
message size less than 250k in my procmailrc I've dropped it to 50k and
will see what happens. I could probably drop some of the extra rulesets
I run also which would probably cut down on access until I can upgrade
my memory. I'm going to consider this thread closed then as being solved
with "upgrade to more memory". Many thanks to all who replied and
offered suggestions. And to the SA Team, keep up the great work!
Chris
--
Chris
KeyID 0xE372A7DA98E6705C
Re: scantime=249.2; scantime=175.0; scantime=190.9; scantime=68.9
Posted by John Hardin <jh...@impsec.org>.
On Sun, 5 Sep 2010, Chris wrote:
> On Sun, 2010-09-05 at 12:33 -0500, Len Conrad wrote:
>>> Mem: 772880k total, 685316k used, 87564k free, 31344k buffers
>>> Swap: 1076312k total, 249032k used, 827280k free, 156328k cached
>>
>> 250MB swapped, for less than 1 GB RAM, used is disastrous for an MTA.
>>
>> Increase RAM to 2GB, or until swap is always "0k used"
>
> It's just a single user home system. True, I probably do need to
> increase ram but I 'don't' think this has a bearing on this issue though
> I may be wrong.
Your system is swapping. That kills performance, pretty much across the
board. Either buy more memory or accept the impact of an underprovisioned
machine on the performance of mail delivery.
Do you really need your mail to be delivered _that_ promptly? If
interactive performance is acceptable then what does it matter if an email
(delivered in the background) takes 30 seconds or 300 seconds to be
stuffed into your inbox?
How many email users is this supporting? You might want to consider some
glue-level performance hacks - for example, if your box is supporting
email for your family members, and all delivery for that is local, then
you could configure your MTA to not pass locally-originated-and-destined
messages through clamav or SA at all.
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
Sheep have only two speeds: graze and stampede. -- LTC Grossman
-----------------------------------------------------------------------
12 days until the 223rd anniversary of the signing of the U.S. Constitution
Re: scantime=249.2; scantime=175.0; scantime=190.9; scantime=68.9
Posted by Chris <cp...@embarqmail.com>.
On Sun, 2010-09-05 at 12:33 -0500, Len Conrad wrote:
> >Mem: 772880k total, 685316k used, 87564k free, 31344k buffers
> >Swap: 1076312k total, 249032k used, 827280k free, 156328k cached
>
> 250MB swapped, for less than 1 GB RAM, used is disastrous for an MTA.
>
> Increase RAM to 2GB, or until swap is always "0k used"
>
> Len
>
It's just a single user home system. True, I probably do need to
increase ram but I 'don't' think this has a bearing on this issue though
I may be wrong.
--
Chris
KeyID 0xE372A7DA98E6705C