You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Ron Smith <po...@pmbx.net> on 2008/07/23 14:15:04 UTC
Memory Leak?
Since upgrading to SA 3.2.5 on
Mac OS 10.5.4 with
perl5 (revision 5 version 8 subversion 8) and
mysql ver 14.12 Distrib 5.0.51b I've seen a serious memory leak.
Over the course of 16 hours or so while looking at the Activity viewer
I watch an ever-increasing inactive memory. I have 4 gigs of ram in my
PowerMac G5 as well. At some point all mail movement in SA stalls.
Anyone else seeing this?
Ron Smith
postmaster@pmbx.net
"Having an email problem is painful, but character-building."
Re: Memory Leak?
Posted by Ron Smith <po...@pmbx.net>.
Thanks so much for your kind response, Jamie. It is much appreciated!
Here's the issue which started about 2 or 3 weeks ago.
Updates/upgrades performed about the same time:
OS X 10.4.11 to OS X 10.5.3/10.5.4
SA from 3.1.8 to 3.2.5
cgpav/clamav... went from 0.6x to 0.8x, but it is currently turned off.
CommuniGate Pro 5.2.2 to 5.2.5
Configuration changes:
stopped using the xbl.spamhaus.org blocklists, spam load increased
At first I was concerned when looking that the Activity Viewer (in the
Utilities folder on all Macs) that I could see the issue. The inactive
memory rises to the point that the free memory is a small sliver of
the pie. Apple's knowledgebase indicates that there should not be an
issue with this and indicates this is reused on demand and is really a
cache. I am having my doubts however.
Processing slows and email flows like syrup through a straw at this
point. Seems like spamd processing is taking many minutes. Using the -
L option to turn off DNS checks seems to make no difference.
Reinstituting the server blocklists sees the server load fall. Mail
continues to be delivered as server load increases but the number of
'orpahaned' .tmp files increases. These are fully processed spamed
email ready for submission, but it seems the spamd exceeds the
CommuniGate pro 2 minute limit for a shell call. Stalker Software,
makers of CG Pro, don't support SA so they simply reply that if its
taking that long for SA to process spam, then I should consider their
Mailshell or Cloudmark plugin which can process 20 emails per second.
(Mailshell did a poor job when I used it several years ago, so speed
isn't everything. SA does a much better job at identifying spam I
think.)
I have not ever used sa-compile, but am going to get that installed
this weekend to see if I can bypass the Perl-SA interpretation of
local regex rules (I know it won't compile all the rules though) and
improve performance and help see if this is possibly the source of the
problem.
I understand that this may sound obscure, but believe me, I'm not a
novice and there seems to be a real issue. I've considered even
dropping back to OS 10.4.11, a time-consuming but possibly necessary
step in further identifying this issue.
Ron Smith
postmaster@pmbx.net
"Having an email problem is painful, but character-building."
On Jul 25, 2008, at 9:16 AM, James Pratt wrote:
>> -----Original Message-----
>> From: Ron Smith [mailto:postmaster@pmbx.net]
>> Sent: Friday, July 25, 2008 8:31 AM
>> To: users@spamassassin.apache.org
>> Subject: Re: Memory Leak?
>>
>> Yes, Kai, I noticed there were no replies. In fact, I've noticed that
>> for the most part many of the folks on this list try to be as
>> unhelpful as possible to new posters on the list. Almost like there
>> is
>> challenge over who can be the quickest on either a rude comment or
> put-
>> down.
>
> Wow... Personally, I find this list to be one of the most helpful,
> least-rude lists around... I've only been here about 5 years too.
>
> I missed the prior emails on this thread, but if you can detail more
> about your setup perhaps someone with the same setup (issues or not?)
> may contact you ...
>
> To be honest, it's kind of a "bad time" for many admins - DNS servers
> around the world need to be patched ASAP, and there are a LOT of them
> out there that are vulnerable to the latest (real, and very bad!)
> cache-poisoning exploit. If you run a DNS server, and have not yet
> tested/patched it, *please* see
> http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2008-1447 asap.
>
> Regards,
> jamie
>
>
RE: Memory Leak?
Posted by James Pratt <jp...@norwich.edu>.
> -----Original Message-----
> From: Ron Smith [mailto:postmaster@pmbx.net]
> Sent: Friday, July 25, 2008 8:31 AM
> To: users@spamassassin.apache.org
> Subject: Re: Memory Leak?
>
> Yes, Kai, I noticed there were no replies. In fact, I've noticed that
> for the most part many of the folks on this list try to be as
> unhelpful as possible to new posters on the list. Almost like there is
> challenge over who can be the quickest on either a rude comment or
put-
> down.
Wow... Personally, I find this list to be one of the most helpful,
least-rude lists around... I've only been here about 5 years too.
I missed the prior emails on this thread, but if you can detail more
about your setup perhaps someone with the same setup (issues or not?)
may contact you ...
To be honest, it's kind of a "bad time" for many admins - DNS servers
around the world need to be patched ASAP, and there are a LOT of them
out there that are vulnerable to the latest (real, and very bad!)
cache-poisoning exploit. If you run a DNS server, and have not yet
tested/patched it, *please* see
http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2008-1447 asap.
Regards,
jamie
Re: Memory Leak?
Posted by Matus UHLAR - fantomas <uh...@fantomas.sk>.
> On Jul 25, 2008, at 4:31 AM, Kai Schaetzl wrote:
> >You notice that you didn't answer the question about memory? Until now
> >you haven't verified at all that the problem is connected to sa.
On 25.07.08 08:31, Ron Smith wrote:
> Yes, Kai, I noticed there were no replies. In fact,
You were expected to reply. You did not. How can anyone help you if you
don't give him enough informations?
> I've noticed that for the most part many of the folks on this list try to
> be as unhelpful as possible to new posters on the list. Almost like there
> is challenge over who can be the quickest on either a rude comment or put-
> down.
You won't get any help with such attitude.
--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
99 percent of lawyers give the rest a bad name.
Re: Memory Leak?
Posted by jdow <jd...@earthlink.net>.
Ron, you still don't seem to indicate that you've figured out there is
no memory leak.
Rather than bothering to berate Kai at this point you might modify
the way spamd is started to reduce the number of children. That is a
sure way to run out of memory and drive the system into swap space.
Once it goes into swap space nothing fails, per se; but, throughput
goes to perdition. Reducing the memory load on the machine will VERY
much help the speed of each SpamAssassin run.
If you have high email throughput demands you may want to reduce the
number of rule sets you are loading if you are using any Rules du Jour
rules from the SpamAssassin Rules Emporium. Reduce the number of other
programs running on your SpamAssassin machine. And of course, reduce
the number of spamd children that are running if you cannot add some
significant amounts of memory.
Memory is important. I run a very small system with limited demands.
So I run a lot of SARE rule sets. Spamd uses about 50 megabytes for
the rule sets. I also use per user rules. That increases the overall
memory footprint. At one time I ran on a 256 megabyte machine that
had some other modest demands placed upon it. When I ran more than
4 children the system slowed down by a very large factor. I ran out
of enough memory to keep the entire array of immediately active
software within RAM.
(If SpamAssassin leaked memory you can bet I'd have been exquisitly
abusive to the person's responsible. I'd notice it remarkably rapidly
because I do tend to watch for that sort of thing. And I beat myself
to death over leaks I create from time to time. So at least I am fair
when I beat up others I catch with a leak. The other folks here are
probably even more sensitive than I am, like the fellow who was
running SA on a system that handled more traffic than AOL ever has
handled. He was an extreme case. But there are some big system admins
on this list. And they'd complain instantly. With that in mind I might
note that your choice of subject was "sub-optimal". It raises the
"BS" flag or the "Troll" flag, rightfully or wrongfully. "Memory
problems" might have been more diplomatic.)
{^_^} Joanne (ME lecturing about diplomacy? You GOT to be kidding!)
----- Original Message -----
From: "Ron Smith" <po...@pmbx.net>
Sent: Friday, 2008, July 25 08:11
Kai, I tried to explain that I didn't get that email response. We have
been having mail delivery issues as a result of this problem. I was
asking for help, not a scolding. You assume wrongly that because you
didn't get a reply that I ignored you... I didn't get your reponse.
Ron Smith
postmaster@pmbx.net
"Having an email problem is painful, but character-building."
On Jul 25, 2008, at 10:31 AM, Kai Schaetzl wrote:
> Ron, what are you up? You claim that there is a memory issue with SA.
> Fine, could be so. You where asked to provide memory details. You didn't.
> I told you you didn't. You snap at me and still don't provide the memory
> details. Tells me you are not really interested in investigating the
> issue
> and not really interested in reading replies.
>
> Kai
>
> --
> Kai Schätzl, Berlin, Germany
> Get your web at Conactive Internet Services: http://www.conactive.com
>
>
>
Re: Memory Leak?
Posted by Ron Smith <po...@pmbx.net>.
Hello, Kai. Your comments are much appreciated.
Ron Smith
postmaster@pmbx.net
"Having an email problem is painful, but character-building."
On Jul 25, 2008, at 2:31 PM, Kai Schaetzl wrote:
> Sure you got that reply. You quoted his mail. But you didn't answer
> the
> question about memory. That's all I pointed out - and got this snap
> back.
>
> Ron, in case you didn't understand at all: you claim there is a memory
> issue. But you have not provided any stats to date that back this
> claim. A
> reduction in your "inactive memory" (whatever that is) is no proof for
You are correct that inactive memory is not supposed to act that way.
But I do observe that it reaches a point where it does not appear that
inactive memory is being reused like its supposed and the way Apple
describes.
>
> nothing at all. What you want to do is check the memory usage of
> *spamd*
Yes, I've observed that and it does seem to stay nearly the same but
the processes become sluggish and sluggish until they are processing
only one or two emails a minute.
>
> and related software and provide figures how big it is (absolute and
> relative), how it grows etc. And thinking about this "inactive memory"
> thing I think you should do some research and get information (also
> to us)
Here's Apples URI on inactive memory: http://support.apple.com/kb/HT1342
> what this actually means. Without any knowledge about the naming
> conventions on your Mac OS a growing "inactive memory" would
> actually mean
> something *positive* to me as it might indicate that less memory is
> in use
> than before. That's a good thing, isn't it? That naming may be a
According the doc it should be a good thing. But when it tops off, it
doesn't seem to behave the way they say or describe.
Activity Monitor is a process monitoring app in the Utilities folder
of every OS X installation. You can see processes, memory allocations,
cpu times for each etcetera. That inactive memory pie chart is part of
that and it breaks down the memory allocation.
My other OS X servers are not have the ever increasing inactive memory
that I'm seeing with the mail server.
>
> misleading trap of your Mac OS. So, you should explain it. And that
> "Activity viewer" (again, whatever that is) may give inaccurate
> figures,
> anyway. You need to get some figures from command-line tools.
>
> Kai
Re: Memory Leak?
Posted by Kai Schaetzl <ma...@conactive.com>.
Ron Smith wrote on Fri, 25 Jul 2008 11:11:32 -0400:
> Kai, I tried to explain that I didn't get that email response.
Sure you got that reply. You quoted his mail. But you didn't answer the
question about memory. That's all I pointed out - and got this snap back.
Ron, in case you didn't understand at all: you claim there is a memory
issue. But you have not provided any stats to date that back this claim. A
reduction in your "inactive memory" (whatever that is) is no proof for
nothing at all. What you want to do is check the memory usage of *spamd*
and related software and provide figures how big it is (absolute and
relative), how it grows etc. And thinking about this "inactive memory"
thing I think you should do some research and get information (also to us)
what this actually means. Without any knowledge about the naming
conventions on your Mac OS a growing "inactive memory" would actually mean
something *positive* to me as it might indicate that less memory is in use
than before. That's a good thing, isn't it? That naming may be a
misleading trap of your Mac OS. So, you should explain it. And that
"Activity viewer" (again, whatever that is) may give inaccurate figures,
anyway. You need to get some figures from command-line tools.
Kai
--
Kai Schätzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com
Re: Memory Leak?
Posted by Ron Smith <po...@pmbx.net>.
Kai, I tried to explain that I didn't get that email response. We have
been having mail delivery issues as a result of this problem. I was
asking for help, not a scolding. You assume wrongly that because you
didn't get a reply that I ignored you... I didn't get your reponse.
Ron Smith
postmaster@pmbx.net
"Having an email problem is painful, but character-building."
On Jul 25, 2008, at 10:31 AM, Kai Schaetzl wrote:
> Ron, what are you up? You claim that there is a memory issue with SA.
> Fine, could be so. You where asked to provide memory details. You
> didn't.
> I told you you didn't. You snap at me and still don't provide the
> memory
> details. Tells me you are not really interested in investigating the
> issue
> and not really interested in reading replies.
>
> Kai
>
> --
> Kai Schätzl, Berlin, Germany
> Get your web at Conactive Internet Services: http://www.conactive.com
>
>
>
Re: Memory Leak?
Posted by Kai Schaetzl <ma...@conactive.com>.
Ron, what are you up? You claim that there is a memory issue with SA.
Fine, could be so. You where asked to provide memory details. You didn't.
I told you you didn't. You snap at me and still don't provide the memory
details. Tells me you are not really interested in investigating the issue
and not really interested in reading replies.
Kai
--
Kai Schätzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com
Re: Memory Leak?
Posted by jdow <jd...@earthlink.net>.
> What parameters do you use to start spamd?
"Nevermind" you just told us.
It might help if you used the > and < keys in top to sort by %MEM.
These are my top few lines here on a 1 gigabyte machine:
11653 root 20 0 67024 58m 2884 S 0.0 5.7 0:26.75 spamd
19970 named 20 0 88584 50m 2220 S 0.0 5.0 20:30.11 named
11701 root 20 0 56972 48m 2684 S 0.0 4.8 4:15.79 spamd
3202 root 20 0 71068 47m 1172 S 0.0 4.7 16:58.28 clamd.SA
12197 root 20 0 56972 46m 640 S 0.0 4.6 0:00.00 spamd
29400 root 20 0 99.8m 35m 4848 S 0.0 3.5 7:55.31 X
The spamd memory percentage numbers might help us to know.
In the meantime your numbers seem somewhat odd. Unfortunately the
accounting in your activity monitor is useless. It does not tell
what command perl is running that is taking so much memory. It
appears "mdworker", whatever that is, eats a lot. So do some other
odd items which suggest this is a machine servicing far more than
email only.
Also do you have any SpamAssassin Rules Emporium rule sets that are
active? And if so which ones?
Another idea comes to mind, is there some form of "accounting" running
on the machine which might intentionally slow the the machine?
{^_^} Joanne
----- Original Message -----
From: "jdow" <jd...@earthlink.net>
To: <us...@spamassassin.apache.org>
Sent: Friday, 2008, July 25 23:45
Subject: Re: Memory Leak?
> What parameters do you use to start spamd?
>
> {^_^}
> ----- Original Message -----
> From: "Ron Smith" <po...@pmbx.net>
> Sent: Friday, 2008, July 25 13:56
>
>
>> Hi, Robert. No, I've been dealing with this issue now for two solid
>> weeks daily until 2am and its been in my mind every waking moment.
>> Believe me I've been over and over and over it and wouldn't have posed
>> my concerns lightly.
>>
>> One thing is clear. When I turn off viral screening and everything
>> else extraneous... the apparent memory issue continues until I stop
>> spamd. And the problem started when I upgraded from version 3.1.8 to
>> 3.2.5 which I think is when a lot of URI tests were introduced.
>>
>> My current focus is that either its spamd / perl / OS X 10.5.4. I'm
>> trying everything else before I drop back to Tiger which will require
>> no small effort. I have considered even mysql but that's very low on
>> my list.
>>
>> And let me say that I really do appreciate your input. I do need to
>> make sure that I've gone over everything so keep the questions coming
>> and hold my feet to the fire on this till I get it resolved. ;-)
>>
>> Ron Smith
>> postmaster@pmbx.net
>>
>> "Having an email problem is painful, but character-building."
>>
>> On Jul 25, 2008, at 11:30 AM, Robert - elists wrote:
>>
>>> Ron
>>>
>>> You are kinda shooting in the dark on the memory leak thread stuff.
>>>
>>> You need to get a lil deeper in the system(s) to know.
>>>
>>> I don't know what programming tools you are familiar with yet you
>>> could do
>>> some tracing and know exactly what is going on and when it is
>>> happening and
>>> then you can immediately correct or work on determining why and
>>> correct
>>>
>>> - rh
>>>
>>>
Re: Memory Leak?
Posted by jdow <jd...@earthlink.net>.
What parameters do you use to start spamd?
{^_^}
----- Original Message -----
From: "Ron Smith" <po...@pmbx.net>
Sent: Friday, 2008, July 25 13:56
> Hi, Robert. No, I've been dealing with this issue now for two solid
> weeks daily until 2am and its been in my mind every waking moment.
> Believe me I've been over and over and over it and wouldn't have posed
> my concerns lightly.
>
> One thing is clear. When I turn off viral screening and everything
> else extraneous... the apparent memory issue continues until I stop
> spamd. And the problem started when I upgraded from version 3.1.8 to
> 3.2.5 which I think is when a lot of URI tests were introduced.
>
> My current focus is that either its spamd / perl / OS X 10.5.4. I'm
> trying everything else before I drop back to Tiger which will require
> no small effort. I have considered even mysql but that's very low on
> my list.
>
> And let me say that I really do appreciate your input. I do need to
> make sure that I've gone over everything so keep the questions coming
> and hold my feet to the fire on this till I get it resolved. ;-)
>
> Ron Smith
> postmaster@pmbx.net
>
> "Having an email problem is painful, but character-building."
>
> On Jul 25, 2008, at 11:30 AM, Robert - elists wrote:
>
>> Ron
>>
>> You are kinda shooting in the dark on the memory leak thread stuff.
>>
>> You need to get a lil deeper in the system(s) to know.
>>
>> I don't know what programming tools you are familiar with yet you
>> could do
>> some tracing and know exactly what is going on and when it is
>> happening and
>> then you can immediately correct or work on determining why and
>> correct
>>
>> - rh
>>
>>
Re: Memory Leak?
Posted by Ron Smith <po...@pmbx.net>.
Hi, Robert. No, I've been dealing with this issue now for two solid
weeks daily until 2am and its been in my mind every waking moment.
Believe me I've been over and over and over it and wouldn't have posed
my concerns lightly.
One thing is clear. When I turn off viral screening and everything
else extraneous... the apparent memory issue continues until I stop
spamd. And the problem started when I upgraded from version 3.1.8 to
3.2.5 which I think is when a lot of URI tests were introduced.
My current focus is that either its spamd / perl / OS X 10.5.4. I'm
trying everything else before I drop back to Tiger which will require
no small effort. I have considered even mysql but that's very low on
my list.
And let me say that I really do appreciate your input. I do need to
make sure that I've gone over everything so keep the questions coming
and hold my feet to the fire on this till I get it resolved. ;-)
Ron Smith
postmaster@pmbx.net
"Having an email problem is painful, but character-building."
On Jul 25, 2008, at 11:30 AM, Robert - elists wrote:
> Ron
>
> You are kinda shooting in the dark on the memory leak thread stuff.
>
> You need to get a lil deeper in the system(s) to know.
>
> I don't know what programming tools you are familiar with yet you
> could do
> some tracing and know exactly what is going on and when it is
> happening and
> then you can immediately correct or work on determining why and
> correct
>
> - rh
>
>
RE: Memory Leak?
Posted by Robert - elists <li...@abbacomm.net>.
Ron
You are kinda shooting in the dark on the memory leak thread stuff.
You need to get a lil deeper in the system(s) to know.
I don't know what programming tools you are familiar with yet you could do
some tracing and know exactly what is going on and when it is happening and
then you can immediately correct or work on determining why and correct
- rh
Re: Memory Leak?
Posted by Ron Smith <po...@pmbx.net>.
Hi, Martin. That's also a very good suggestion. Now I thought that
using the -L parameter would turn all those off. I did try that
thinking that that was the case however I really wasn't sure that it
made a difference. I'll try your suggestion on the zero scores in the
local.cf too.
We do run named on that same server and in fact run a total of four
nameservers. I'll check and make sure that is the one that is the
first choice though in the System Preferences.
I did think of the this and wondered if there was a DDoS against one
of those servers that might be causing long pauses in responses. Just
about 5 weeks ago I had a DDoS saturating my incoming SMTPs... Despite
that, and without SA running we were able to function. (I'd be glad to
share individually with anyone the details of how we did that if you
are suffering the same.)
Ron Smith
postmaster@pmbx.net
"Having an email problem is painful, but character-building."
On Jul 25, 2008, at 11:12 AM, Martin.Hepworth wrote:
> Ron
>
> I'd check what RBL's and URI-RBL's you are running.
>
> If you haven't turned any of them then you're running them all -
> which can lead to very long processing times.
>
> Choose one or two you want by looking through the 20_dnsbl_tests.cf
> file. And give then rest a zero score in local.cf
>
> Running a local caching nameserver on the machine itself can help
> quite a bit here too.
RE: Memory Leak?
Posted by "Martin.Hepworth" <ma...@solidstatelogic.com>.
Ron
I'd check what RBL's and URI-RBL's you are running.
If you haven't turned any of them then you're running them all - which can lead to very long processing times.
Choose one or two you want by looking through the 20_dnsbl_tests.cf file. And give then rest a zero score in local.cf
Running a local caching nameserver on the machine itself can help quite a bit here too.
--
Martin Hepworth
Snr Systems Administrator
Solid State Logic
Tel: +44 (0)1865 842300
> -----Original Message-----
> From: Ron Smith [mailto:postmaster@pmbx.net]
> Sent: 25 July 2008 15:45
> To: d.hill@yournetplus.com; users@spamassassin.apache.org
> Subject: Re: Memory Leak?
>
> Thanks, Duane for your kind comment. Yes I have tried
> different methods. I used to use cgpsa. cgpav was only for
> antiviral stuff and it failed miserably when I tried to have
> it doing the spamassassin calls also. It seems that no matter
> how I call spamd there is an issue.
>
> I'm using scanspam.sh as an execute call in CGPro. In fact I
> think I've actually identified where in the shell script the
> issue may be occurring. Here's an excerpt from the key part:
>
> /var/CommuniGate/spam/spamprep "$myCgate/$QueuePath"
> "$ReturnPath"
> "$Username" |
> /usr/bin/spamc -d 127.0.0.1 -t 100 -u "$Username" >>
> "$myCgate/ Submitted/$NewFile"
> mv /var/CommuniGate/Submitted/$NewFile
> /var/CommuniGate/Submitted/ $FinalFile
>
> CG Pro gives execute scripts 2 minutes to finish or it kills them.
> Possibly it calls the script again, or so I thought, which
> might be the source for the multiple .tmp extensions and long
> processing times.
> If the first line above is calling the spamd, then CG Pro
> kills the script before the mv command adds the .sub
> extension. That would account for the orphaned .tmp files
> that have the spamd processing finished, but never got submitted.
>
> Still that does not tell me that there is a problem with CG
> Pro and shell scripts. It could likely just as much mean that
> spamd is slowed because of a memory leak in that code.
>
> Notice also that the -t 100, which means that spamd should
> finish processing within 100 seconds (or so I understand)
> SHOULD mean that CGPro shouldn't ever reach the 120 second (2
> minute) limit that would cause it to kill the sh process.
>
> When I watch the submitted folder, MOST of the processing
> however occurs very quickly and there are usually not more
> that 3 or 4 .tmp files being processed. Until the spam load
> increases. That's where my suspicion of a memory leak in
> spamd comes from.
>
> In order to further test this, I'm considering altering the
> script above to actually call another script basically
> containing the two lines above, thereby preventing CGPro from
> killing the script prematurely. That's a next step. If the
> delay in processing is still present then, I would think that
> I've moved suspicion away from CG Pro/ spamd interaction as a
> cause for this.
>
> Ron Smith
> postmaster@pmbx.net
>
> "Having an email problem is painful, but character-building."
>
> On Jul 25, 2008, at 9:46 AM, Duane Hill wrote:
>
> > For a test, ever thought about changing to a different script?
> >
> > Even through we have two Postfix border servers doing filtering, I
> > still have something on our internal (antiquated)
> CommuniGate server.
> >
> > I am using SA 3.2.5 with cgpsa and there hasn't been one bit of an
> > issue.
> >
> > http://www.tffenterprises.com/cgpsa/
> >
> > -d
>
>
**********************************************************************
Confidentiality : This e-mail and any attachments are intended for the
addressee only and may be confidential. If they come to you in error
you must take no action based on them, nor must you copy or show them
to anyone. Please advise the sender by replying to this e-mail
immediately and then delete the original from your computer.
Opinion : Any opinions expressed in this e-mail are entirely those of
the author and unless specifically stated to the contrary, are not
necessarily those of the author's employer.
Security Warning : Internet e-mail is not necessarily a secure
communications medium and can be subject to data corruption. We advise
that you consider this fact when e-mailing us.
Viruses : We have taken steps to ensure that this e-mail and any
attachments are free from known viruses but in keeping with good
computing practice, you should ensure that they are virus free.
Red Lion 49 Ltd T/A Solid State Logic
Registered as a limited company in England and Wales
(Company No:5362730)
Registered Office: 25 Spring Hill Road, Begbroke, Oxford OX5 1RU,
United Kingdom
**********************************************************************
Re: Memory Leak?
Posted by Ron Smith <po...@pmbx.net>.
Thanks, Duane for your kind comment. Yes I have tried different
methods. I used to use cgpsa. cgpav was only for antiviral stuff and
it failed miserably when I tried to have it doing the spamassassin
calls also. It seems that no matter how I call spamd there is an issue.
I'm using scanspam.sh as an execute call in CGPro. In fact I think
I've actually identified where in the shell script the issue may be
occurring. Here's an excerpt from the key part:
/var/CommuniGate/spam/spamprep "$myCgate/$QueuePath" "$ReturnPath"
"$Username" |
/usr/bin/spamc -d 127.0.0.1 -t 100 -u "$Username" >> "$myCgate/
Submitted/$NewFile"
mv /var/CommuniGate/Submitted/$NewFile /var/CommuniGate/Submitted/
$FinalFile
CG Pro gives execute scripts 2 minutes to finish or it kills them.
Possibly it calls the script again, or so I thought, which might be
the source for the multiple .tmp extensions and long processing times.
If the first line above is calling the spamd, then CG Pro kills the
script before the mv command adds the .sub extension. That would
account for the orphaned .tmp files that have the spamd processing
finished, but never got submitted.
Still that does not tell me that there is a problem with CG Pro and
shell scripts. It could likely just as much mean that spamd is slowed
because of a memory leak in that code.
Notice also that the -t 100, which means that spamd should finish
processing within 100 seconds (or so I understand) SHOULD mean that
CGPro shouldn't ever reach the 120 second (2 minute) limit that would
cause it to kill the sh process.
When I watch the submitted folder, MOST of the processing however
occurs very quickly and there are usually not more that 3 or 4 .tmp
files being processed. Until the spam load increases. That's where my
suspicion of a memory leak in spamd comes from.
In order to further test this, I'm considering altering the script
above to actually call another script basically containing the two
lines above, thereby preventing CGPro from killing the script
prematurely. That's a next step. If the delay in processing is still
present then, I would think that I've moved suspicion away from CG Pro/
spamd interaction as a cause for this.
Ron Smith
postmaster@pmbx.net
"Having an email problem is painful, but character-building."
On Jul 25, 2008, at 9:46 AM, Duane Hill wrote:
> For a test, ever thought about changing to a different script?
>
> Even through we have two Postfix border servers doing filtering, I
> still have something on our internal (antiquated) CommuniGate server.
>
> I am using SA 3.2.5 with cgpsa and there hasn't been one bit of an
> issue.
>
> http://www.tffenterprises.com/cgpsa/
>
> -d
Re: Memory Leak?
Posted by Duane Hill <d....@yournetplus.com>.
On Fri, 25 Jul 2008, Ron Smith wrote:
> When I reinstituted the blocklists in CommuniGate Pro, the spam load falls
> dramatically and SA continues to function. And yet I'm still getting tmp
> files in the Submitted folder that have multiple .tmp extensions as though
> spamd was being respawned by the scanspam.sh script after a child process
> dies inadvertently. I started having these issues as I said when I upgraded
> from 3.1.8 to 3.2.5. I also had installed the OS X 10.5.4 update. I turned
> Clamav off which was being called by cgpav with no change. I've run CG Pro
> even without SA and there is no problem, even without the blocklists. CGPro
> easily can handle the 100,000 plus spam our server gets in a day.
For a test, ever thought about changing to a different script?
Even through we have two Postfix border servers doing filtering, I still
have something on our internal (antiquated) CommuniGate server.
I am using SA 3.2.5 with cgpsa and there hasn't been one bit of an issue.
http://www.tffenterprises.com/cgpsa/
-d
Re: Memory Leak?
Posted by Ron Smith <po...@pmbx.net>.
Yes, Kai, I noticed there were no replies. In fact, I've noticed that
for the most part many of the folks on this list try to be as
unhelpful as possible to new posters on the list. Almost like there is
challenge over who can be the quickest on either a rude comment or put-
down.
I've been using SA for a long, long time and serve several thousand
users who are not local but situated around the world. Don't you think
you might want to know if I noticed an issue between upgrades that
might affect other SA users?
No matter, I've got tough skin and I've just as much right to publish
on the list as anyone else (and I'll probably get flamed over this). I
don't care though and any comebacks will just be ignored. I'm older
and wise enough to not be affected by that schoolyard childishness so
those that want to wast their time, have at it. I'm fast on the delete
key, emotionally immune, and I won't waste my time reading or
responding to jabs or insults.
Now, for those who want to engage is serious, and adult discussion
about what I think is a possible memory issue... I am still concerned
that there is a memory issue with SA. Because I've been aware that
under heavy loads, I've seen SA start to backup and the file count in
the queue rise into the thousands over 5 to 8 hours.
When I reinstituted the blocklists in CommuniGate Pro, the spam load
falls dramatically and SA continues to function. And yet I'm still
getting tmp files in the Submitted folder that have multiple .tmp
extensions as though spamd was being respawned by the scanspam.sh
script after a child process dies inadvertently. I started having
these issues as I said when I upgraded from 3.1.8 to 3.2.5. I also had
installed the OS X 10.5.4 update. I turned Clamav off which was being
called by cgpav with no change. I've run CG Pro even without SA and
there is no problem, even without the blocklists. CGPro easily can
handle the 100,000 plus spam our server gets in a day.
I've been wondering if this could be simply an inherent interpretive
slow-down in Perl. I had not compiled with sa-compile, but plan to do
that today or tomorrow as soon as I get re2c installed to see if there
is a performance improvement. I'll be interested to see how this
affects the multiple .tmp extensions. I'd be very interested to hear
from anyone else who've had issues like these that seemed to be
improved with compilation.
All respectful and cordial replies will be greeted with equal
response. ;-)
Ron Smith
Veteran Pediatrician of 25 years
Harley-Davidson Ultra Glide man (minus tattoos or piercings), and one
tough son-of-a-gun!
postmaster@pmbx.net
"Having an email problem is painful, but character-building."
On Jul 25, 2008, at 4:31 AM, Kai Schaetzl wrote:
> You notice that you didn't answer the question about memory? Until
> now you
> haven't verified at all that the problem is connected to sa.
>
> Kai
>
> --
> Kai Schätzl, Berlin, Germany
> Get your web at Conactive Internet Services: http://www.conactive.com
>
>
>
Re: Memory Leak?
Posted by Kai Schaetzl <ma...@conactive.com>.
You notice that you didn't answer the question about memory? Until now you
haven't verified at all that the problem is connected to sa.
Kai
--
Kai Schätzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com
Re: Memory Leak?
Posted by Ron Smith <po...@pmbx.net>.
I've tried anywhere from 4 to 10 children with the same results. I've
used the scanspam script (which I have for years before now) and even
tried using spamd from the cgpav. Cgpav failed quickly also and at
first I thought that was the issue, but there has been no difference
with the cgpav helper turned off in CommuniGate Pro.
I downgraded from CommuniGate Pro 3.2.5 to 3.2.2 late last night with
no difference either. These problems all have arisen not long after I
upgraded from SA 3.1.8 to 3.2.5 about 2 to 3 weeks ago.
Ron Smith
postmaster@pmbx.net
"Having an email problem is painful, but character-building."
On Jul 23, 2008, at 8:40 AM, Matt Kettler wrote:
> Ron Smith wrote:
>> Since upgrading to SA 3.2.5 on
>>
>> Mac OS 10.5.4 with
>> perl5 (revision 5 version 8 subversion 8) and
>> mysql ver 14.12 Distrib 5.0.51b I've seen a serious memory leak.
>>
>> Over the course of 16 hours or so while looking at the Activity
>> viewer I watch an ever-increasing inactive memory. I have 4 gigs of
>> ram in my PowerMac G5 as well. At some point all mail movement in
>> SA stalls.
>>
>> Anyone else seeing this?
>>
>> Ron Smith
>> postmaster@pmbx.net
>>
>> "Having an email problem is painful, but character-building."
>>
>>
> I've not seen that myself. How many spamd children are running? Is
> their count constantly increasing, or is it the size of each that's
> growing but the count remains constant?
Re: Memory Leak?
Posted by Matt Kettler <mk...@verizon.net>.
Ron Smith wrote:
> Since upgrading to SA 3.2.5 on
>
> Mac OS 10.5.4 with
> perl5 (revision 5 version 8 subversion 8) and
> mysql ver 14.12 Distrib 5.0.51b I've seen a serious memory leak.
>
> Over the course of 16 hours or so while looking at the Activity viewer
> I watch an ever-increasing inactive memory. I have 4 gigs of ram in my
> PowerMac G5 as well. At some point all mail movement in SA stalls.
>
> Anyone else seeing this?
>
> Ron Smith
> postmaster@pmbx.net
>
> "Having an email problem is painful, but character-building."
>
>
I've not seen that myself. How many spamd children are running? Is their
count constantly increasing, or is it the size of each that's growing
but the count remains constant?