Memory Leak?

[Ron Smith-4]

Since upgrading to SA 3.2.5 on

Mac OS 10.5.4 with
perl5 (revision 5 version 8 subversion 8) and
mysql ver 14.12 Distrib 5.0.51b I've seen a serious memory leak.

Over the course of 16 hours or so while looking at the Activity viewer  
I watch an ever-increasing inactive memory. I have 4 gigs of ram in my  
PowerMac G5 as well. At some point all mail movement in SA stalls.

Anyone else seeing this?

Ron Smith
postmaster@...

"Having an email problem is painful, but character-building."

[Matt Kettler-3]

Ron Smith wrote:

> Since upgrading to SA 3.2.5 on
>
> Mac OS 10.5.4 with
> perl5 (revision 5 version 8 subversion 8) and
> mysql ver 14.12 Distrib 5.0.51b I've seen a serious memory leak.
>
> Over the course of 16 hours or so while looking at the Activity viewer
> I watch an ever-increasing inactive memory. I have 4 gigs of ram in my
> PowerMac G5 as well. At some point all mail movement in SA stalls.
>
> Anyone else seeing this?
>
> Ron Smith
> postmaster@...
>
> "Having an email problem is painful, but character-building."
>
>

I've not seen that myself. How many spamd children are running? Is their
count constantly increasing, or is it the size of each that's growing
but the count remains constant?

[Ron Smith-4]

I've tried anywhere from 4 to 10 children with the same results. I've  
used the scanspam script (which I have for years before now) and even  
tried using spamd from the cgpav. Cgpav failed quickly also and at  
first I thought that was the issue, but there has been no difference  
with the cgpav helper turned off in CommuniGate Pro.

I downgraded from CommuniGate Pro 3.2.5 to 3.2.2 late last night with  
no difference either. These problems all have arisen not long after I  
upgraded from SA 3.1.8 to 3.2.5 about 2 to 3 weeks ago.

Ron Smith
postmaster@...

"Having an email problem is painful, but character-building."

On Jul 23, 2008, at 8:40 AM, Matt Kettler wrote:

> Ron Smith wrote:
>> Since upgrading to SA 3.2.5 on
>>
>> Mac OS 10.5.4 with
>> perl5 (revision 5 version 8 subversion 8) and
>> mysql ver 14.12 Distrib 5.0.51b I've seen a serious memory leak.
>>
>> Over the course of 16 hours or so while looking at the Activity  
>> viewer I watch an ever-increasing inactive memory. I have 4 gigs of  
>> ram in my PowerMac G5 as well. At some point all mail movement in  
>> SA stalls.
>>
>> Anyone else seeing this?
>>
>> Ron Smith
>> postmaster@...
>>
>> "Having an email problem is painful, but character-building."
>>
>>
> I've not seen that myself. How many spamd children are running? Is  
> their count constantly increasing, or is it the size of each that's  
> growing but the count remains constant?

[Kai Schaetzl]

You notice that you didn't answer the question about memory? Until now you
haven't verified at all that the problem is connected to sa.

Kai

--
Kai Schätzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com

[Ron Smith-4]

Yes, Kai, I noticed there were no replies. In fact, I've noticed that  
for the most part many of the folks on this list try to be as  
unhelpful as possible to new posters on the list. Almost like there is  
challenge over who can be the quickest on either a rude comment or put-
down.

I've been using SA for a long, long time and serve several thousand  
users who are not local but situated around the world. Don't you think  
you might want to know if I noticed an issue between upgrades that  
might affect other SA users?

No matter, I've got tough skin and I've just as much right to publish  
on the list as anyone else (and I'll probably get flamed over this). I  
don't care though and any comebacks will just be ignored. I'm older  
and wise enough to not be affected by that schoolyard childishness so  
those that want to wast their time, have at it. I'm fast on the delete  
key, emotionally immune, and I won't waste my time reading or  
responding to jabs or insults.


Now, for those who want to engage is serious, and adult discussion  
about what I think is a possible memory issue... I am still concerned  
that there is a memory issue with SA. Because I've been aware that  
under heavy loads, I've seen SA start to backup and the file count in  
the queue rise into the thousands over 5 to 8 hours.

When I reinstituted the blocklists in CommuniGate Pro, the spam load  
falls dramatically and SA continues to function. And yet I'm still  
getting tmp files in the Submitted folder that have multiple .tmp  
extensions as though spamd was being respawned by the scanspam.sh  
script after a child process dies inadvertently. I started having  
these issues as I said when I upgraded from 3.1.8 to 3.2.5. I also had  
installed the OS X 10.5.4 update. I turned Clamav off which was being  
called by cgpav with no change. I've run CG Pro even without SA and  
there is no problem, even without the blocklists. CGPro easily can  
handle the 100,000 plus spam our server gets in a day.

I've been wondering if this could be simply an inherent interpretive  
slow-down in Perl. I had not compiled with sa-compile, but plan to do  
that today or tomorrow as soon as I get re2c installed to see if there  
is a performance improvement. I'll be interested to see how this  
affects the multiple .tmp extensions. I'd be very interested to hear  
from anyone else who've had issues like these that seemed to be  
improved with compilation.

All respectful and cordial replies will be greeted with equal  
response. ;-)

Ron Smith
Veteran Pediatrician of 25 years
Harley-Davidson Ultra Glide man (minus tattoos or piercings), and one  
tough son-of-a-gun!
postmaster@...

"Having an email problem is painful, but character-building."

On Jul 25, 2008, at 4:31 AM, Kai Schaetzl wrote:

> You notice that you didn't answer the question about memory? Until  
> now you
> haven't verified at all that the problem is connected to sa.
>
> Kai
>
> --
> Kai Schätzl, Berlin, Germany
> Get your web at Conactive Internet Services: http://www.conactive.com
>
>
>

[Matus UHLAR - fantomas]

> On Jul 25, 2008, at 4:31 AM, Kai Schaetzl wrote:
> >You notice that you didn't answer the question about memory? Until now
> >you haven't verified at all that the problem is connected to sa.

On 25.07.08 08:31, Ron Smith wrote:
> Yes, Kai, I noticed there were no replies. In fact,

You were expected to reply. You did not. How can anyone help you if you
don't give him enough informations?

> I've noticed that for the most part many of the folks on this list try to
> be as unhelpful as possible to new posters on the list. Almost like there
> is challenge over who can be the quickest on either a rude comment or put-
> down.

You won't get any help with such attitude.
--
Matus UHLAR - fantomas, uhlar@... ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
99 percent of lawyers give the rest a bad name.

[James Pratt-2]

> -----Original Message-----
> From: Ron Smith [mailto:postmaster@...]
> Sent: Friday, July 25, 2008 8:31 AM
> To: users@...
> Subject: Re: Memory Leak?
>
> Yes, Kai, I noticed there were no replies. In fact, I've noticed that
> for the most part many of the folks on this list try to be as
> unhelpful as possible to new posters on the list. Almost like there is
> challenge over who can be the quickest on either a rude comment or

put-
> down.

Wow... Personally, I find this list to be one of the most helpful,
least-rude lists around... I've only been here about 5 years too.

I missed the prior emails on this thread, but if you can detail more
about your setup perhaps someone with the same setup (issues or not?)
may contact you ...

To be honest, it's kind of a "bad time" for many admins - DNS servers
around the world need to be patched ASAP, and there are a LOT of them
out there that are vulnerable to the latest (real, and very bad!)
cache-poisoning exploit.  If you run a DNS server, and have not yet
tested/patched it, *please* see
http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2008-1447 asap.

Regards,
jamie
 

[Bowie Bailey]

Ron Smith wrote:

>
> Now, for those who want to engage is serious, and adult discussion
> about what I think is a possible memory issue... I am still concerned
> that there is a memory issue with SA. Because I've been aware that
> under heavy loads, I've seen SA start to backup and the file count in
> the queue rise into the thousands over 5 to 8 hours.
>
> When I reinstituted the blocklists in CommuniGate Pro, the spam load
> falls dramatically and SA continues to function. And yet I'm still
> getting tmp files in the Submitted folder that have multiple .tmp
> extensions as though spamd was being respawned by the scanspam.sh
> script after a child process dies inadvertently. I started having
> these issues as I said when I upgraded from 3.1.8 to 3.2.5. I also had
> installed the OS X 10.5.4 update. I turned Clamav off which was being
> called by cgpav with no change. I've run CG Pro even without SA and
> there is no problem, even without the blocklists. CGPro easily can
> handle the 100,000 plus spam our server gets in a day.
>
> I've been wondering if this could be simply an inherent interpretive
> slow-down in Perl. I had not compiled with sa-compile, but plan to do
> that today or tomorrow as soon as I get re2c installed to see if there
> is a performance improvement. I'll be interested to see how this
> affects the multiple .tmp extensions. I'd be very interested to hear
> from anyone else who've had issues like these that seemed to be
> improved with compilation.

I don't know much about how SA handles memory internally, but I do know
that it tends to be a memory hog.  This is at least partly due to
optimizing for speed rather than memory use.

The only time I've seen anything like what you are describing is when I
had too many child processes running and the machine went into swap.
Limit the child processes so that the machine doesn't start swapping
under heavy load.  Any usage of swap by SA will seriously slow down your
spam processing.

I don't know about the .tmp files.  That would be a CommuniGate issue.

--
Bowie

[Duane Hill-2]

On Fri, 25 Jul 2008, Ron Smith wrote:

> When I reinstituted the blocklists in CommuniGate Pro, the spam load falls
> dramatically and SA continues to function. And yet I'm still getting tmp
> files in the Submitted folder that have multiple .tmp extensions as though
> spamd was being respawned by the scanspam.sh script after a child process
> dies inadvertently. I started having these issues as I said when I upgraded
> from 3.1.8 to 3.2.5. I also had installed the OS X 10.5.4 update. I turned
> Clamav off which was being called by cgpav with no change. I've run CG Pro
> even without SA and there is no problem, even without the blocklists. CGPro
> easily can handle the 100,000 plus spam our server gets in a day.

For a test, ever thought about changing to a different script?

Even through we have two Postfix border servers doing filtering, I still
have something on our internal (antiquated) CommuniGate server.

I am using SA 3.2.5 with cgpsa and there hasn't been one bit of an issue.

http://www.tffenterprises.com/cgpsa/

-d

[Ron Smith-4]

Thanks so much for your kind response, Jamie. It is much appreciated!  
Here's the issue which started about 2 or 3 weeks ago.

Updates/upgrades performed about the same time:

OS X 10.4.11 to OS X 10.5.3/10.5.4
SA from 3.1.8 to 3.2.5
cgpav/clamav... went from 0.6x to 0.8x, but it is currently turned off.
CommuniGate Pro 5.2.2 to 5.2.5

Configuration changes:
stopped using the xbl.spamhaus.org blocklists, spam load increased

At first I was concerned when looking that the Activity Viewer (in the  
Utilities folder on all Macs) that I could see the issue. The inactive  
memory rises to the point that the free memory is a small sliver of  
the pie. Apple's knowledgebase indicates that there should not be an  
issue with this and indicates this is reused on demand and is really a  
cache. I am having my doubts however.

Processing slows and email flows like syrup through a straw at this  
point. Seems like spamd processing is taking many minutes. Using the -
L option to turn off DNS checks seems to make no difference.

Reinstituting the server blocklists sees the server load fall. Mail  
continues to be delivered as server load increases but the number of  
'orpahaned' .tmp files increases. These are fully processed spamed  
email ready for submission, but it seems the spamd exceeds the  
CommuniGate pro 2 minute limit for a shell call. Stalker Software,  
makers of CG Pro, don't support SA so they simply reply that if its  
taking that long for SA to process spam, then I should consider their  
Mailshell or Cloudmark plugin which can process 20 emails per second.  
(Mailshell did a poor job when I used it several years ago, so speed  
isn't everything. SA does a much better job at identifying spam I  
think.)

I have not ever used sa-compile, but am going to get that installed  
this weekend to see if I can bypass the Perl-SA interpretation of  
local regex rules (I know it won't compile all the rules though) and  
improve performance and help see if this is possibly the source of the  
problem.

I understand that this may sound obscure, but believe me, I'm not a  
novice and there seems to be a real issue. I've considered even  
dropping back to OS 10.4.11, a time-consuming but possibly necessary  
step in further identifying this issue.

Ron Smith
postmaster@...

"Having an email problem is painful, but character-building."

On Jul 25, 2008, at 9:16 AM, James Pratt wrote:

>> -----Original Message-----
>> From: Ron Smith [mailto:postmaster@...]
>> Sent: Friday, July 25, 2008 8:31 AM
>> To: users@...
>> Subject: Re: Memory Leak?
>>
>> Yes, Kai, I noticed there were no replies. In fact, I've noticed that
>> for the most part many of the folks on this list try to be as
>> unhelpful as possible to new posters on the list. Almost like there  
>> is
>> challenge over who can be the quickest on either a rude comment or
> put-
>> down.
>
> Wow... Personally, I find this list to be one of the most helpful,
> least-rude lists around... I've only been here about 5 years too.
>
> I missed the prior emails on this thread, but if you can detail more
> about your setup perhaps someone with the same setup (issues or not?)
> may contact you ...
>
> To be honest, it's kind of a "bad time" for many admins - DNS servers
> around the world need to be patched ASAP, and there are a LOT of them
> out there that are vulnerable to the latest (real, and very bad!)
> cache-poisoning exploit.  If you run a DNS server, and have not yet
> tested/patched it, *please* see
> http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2008-1447 asap.
>
> Regards,
> jamie
>
>

[Ron Smith-4]

Thanks so much for your kind response,  Bowie. That has become my  
conclusion also. Do you use sa-compile and if so, have you seen that  
improve memory usage?

I've tried adjusting the child processes up and down with little  
effect I think. Are you using 2 or 3? What's your timeout setting for  
spamd?

Ron Smith
postmaster@...

"Having an email problem is painful, but character-building."

On Jul 25, 2008, at 9:17 AM, Bowie Bailey wrote:

> I don't know much about how SA handles memory internally, but I do  
> know
> that it tends to be a memory hog.  This is at least partly due to
> optimizing for speed rather than memory use.
>
> The only time I've seen anything like what you are describing is  
> when I
> had too many child processes running and the machine went into swap.
> Limit the child processes so that the machine doesn't start swapping
> under heavy load.  Any usage of swap by SA will seriously slow down  
> your
> spam processing.
>
> I don't know about the .tmp files.  That would be a CommuniGate issue.

[Kai Schaetzl]

Ron, what are you up? You claim that there is a memory issue with SA.
Fine, could be so. You where asked to provide memory details. You didn't.
I told you you didn't. You snap at me and still don't provide the memory
details. Tells me you are not really interested in investigating the issue
and not really interested in reading replies.

Kai

--
Kai Schätzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com

[Ron Smith-4]

Thanks, Duane for your kind comment. Yes I have tried different  
methods. I used to use cgpsa. cgpav was only for antiviral stuff and  
it failed miserably when I tried to have it doing the spamassassin  
calls also. It seems that no matter how I call spamd there is an issue.

I'm using scanspam.sh as an execute call in CGPro. In fact I think  
I've actually identified where in the shell script the issue may be  
occurring. Here's an excerpt from the key part:

   /var/CommuniGate/spam/spamprep "$myCgate/$QueuePath" "$ReturnPath"  
"$Username" |
     /usr/bin/spamc -d 127.0.0.1 -t 100 -u "$Username" >> "$myCgate/
Submitted/$NewFile"
   mv /var/CommuniGate/Submitted/$NewFile /var/CommuniGate/Submitted/
$FinalFile

CG Pro gives execute scripts 2 minutes to finish or it kills them.  
Possibly it calls the script again, or so I thought, which might be  
the source for the multiple .tmp extensions and long processing times.  
If the first line above is calling the spamd, then CG Pro kills the  
script before the mv command adds the .sub extension. That would  
account for the orphaned .tmp files that have the spamd processing  
finished, but never got submitted.

Still that does not tell me that there is a problem with CG Pro and  
shell scripts. It could likely just as much mean that spamd is slowed  
because of a memory leak in that code.

Notice also that the -t 100, which means that spamd should finish  
processing within 100 seconds (or so I understand) SHOULD mean that  
CGPro shouldn't ever reach the 120 second (2 minute) limit that would  
cause it to kill the sh process.

When I watch the submitted folder, MOST of the processing however  
occurs very quickly and there are usually not more that 3 or 4 .tmp  
files being processed. Until the spam load increases. That's where my  
suspicion of a memory leak in spamd comes from.

In order to further test this, I'm considering altering the script  
above to actually call another script basically containing the two  
lines above, thereby preventing CGPro from killing the script  
prematurely. That's a next step. If the delay in processing is still  
present then, I would think that I've moved suspicion away from CG Pro/
spamd interaction as a cause for this.

Ron Smith
postmaster@...

"Having an email problem is painful, but character-building."

On Jul 25, 2008, at 9:46 AM, Duane Hill wrote:

> For a test, ever thought about changing to a different script?
>
> Even through we have two Postfix border servers doing filtering, I  
> still have something on our internal (antiquated) CommuniGate server.
>
> I am using SA 3.2.5 with cgpsa and there hasn't been one bit of an  
> issue.
>
> http://www.tffenterprises.com/cgpsa/
>
> -d

[Bowie Bailey]

Ron Smith wrote:
> Thanks so much for your kind response,  Bowie. That has become my
> conclusion also. Do you use sa-compile and if so, have you seen that
> improve memory usage?

I attempted to use sa-compile, but I kept running into problems, so I
dropped it.  I think the problems were related to an account that has
some user-defined rules.

> I've tried adjusting the child processes up and down with little
> effect I think. Are you using 2 or 3? What's your timeout setting for
> spamd?

I am running spamd like this:

/usr/bin/spamd -d -m 4 --max-conn-per-child=50 -timeout-child=180 -u
mailuser

That's a maximum of 4 spamd child processes and they restart after 50
connections each.  The processes are using 60-65M each.

Drop your maximum number of children until the system runs without using
swap.  You could also drop any unnecessary 3rd party rulesets to lower
the memory consumption of each process.  If you can't keep it out of
swap, you should consider adding some memory.  My system has 1GB of ram,
but the only things running are the mailserver, spamd, clamd, and named.
I have some swap space available, but it is never used.

--
Bowie

[Ron Smith-4]

Kai, I tried to explain that I didn't get that email response. We have  
been having mail delivery issues as a result of this problem. I was  
asking for help, not a scolding. You assume wrongly that because you  
didn't get a reply that I ignored you... I didn't get your reponse.

Ron Smith
postmaster@...

"Having an email problem is painful, but character-building."

On Jul 25, 2008, at 10:31 AM, Kai Schaetzl wrote:

> Ron, what are you up? You claim that there is a memory issue with SA.
> Fine, could be so. You where asked to provide memory details. You  
> didn't.
> I told you you didn't. You snap at me and still don't provide the  
> memory
> details. Tells me you are not really interested in investigating the  
> issue
> and not really interested in reading replies.
>
> Kai
>
> --
> Kai Schätzl, Berlin, Germany
> Get your web at Conactive Internet Services: http://www.conactive.com
>
>
>

[Martin.Hepworth]

Ron

I'd check what RBL's and URI-RBL's you are running.

If you haven't turned any of them then you're running them all - which can lead to very long processing times.

Choose one or two you want by looking through the 20_dnsbl_tests.cf file. And give then rest a zero score in local.cf

Running a local caching nameserver on the machine itself can help quite a bit here too.

--
Martin Hepworth
Snr Systems Administrator
Solid State Logic
Tel: [phone]

> -----Original Message-----
> From: Ron Smith [mailto:postmaster@...]
> Sent: 25 July 2008 15:45
> To: d.hill@...; users@...
> Subject: Re: Memory Leak?
>
> Thanks, Duane for your kind comment. Yes I have tried
> different methods. I used to use cgpsa. cgpav was only for
> antiviral stuff and it failed miserably when I tried to have
> it doing the spamassassin calls also. It seems that no matter
> how I call spamd there is an issue.
>
> I'm using scanspam.sh as an execute call in CGPro. In fact I
> think I've actually identified where in the shell script the
> issue may be occurring. Here's an excerpt from the key part:
>
>    /var/CommuniGate/spam/spamprep "$myCgate/$QueuePath"
> "$ReturnPath"
> "$Username" |
>      /usr/bin/spamc -d 127.0.0.1 -t 100 -u "$Username" >>
> "$myCgate/ Submitted/$NewFile"
>    mv /var/CommuniGate/Submitted/$NewFile
> /var/CommuniGate/Submitted/ $FinalFile
>
> CG Pro gives execute scripts 2 minutes to finish or it kills them.
> Possibly it calls the script again, or so I thought, which
> might be the source for the multiple .tmp extensions and long
> processing times.
> If the first line above is calling the spamd, then CG Pro
> kills the script before the mv command adds the .sub
> extension. That would account for the orphaned .tmp files
> that have the spamd processing finished, but never got submitted.
>
> Still that does not tell me that there is a problem with CG
> Pro and shell scripts. It could likely just as much mean that
> spamd is slowed because of a memory leak in that code.
>
> Notice also that the -t 100, which means that spamd should
> finish processing within 100 seconds (or so I understand)
> SHOULD mean that CGPro shouldn't ever reach the 120 second (2
> minute) limit that would cause it to kill the sh process.
>
> When I watch the submitted folder, MOST of the processing
> however occurs very quickly and there are usually not more
> that 3 or 4 .tmp files being processed. Until the spam load
> increases. That's where my suspicion of a memory leak in
> spamd comes from.
>
> In order to further test this, I'm considering altering the
> script above to actually call another script basically
> containing the two lines above, thereby preventing CGPro from
> killing the script prematurely. That's a next step. If the
> delay in processing is still present then, I would think that
> I've moved suspicion away from CG Pro/ spamd interaction as a
> cause for this.
>
> Ron Smith
> postmaster@...
>
> "Having an email problem is painful, but character-building."
>
> On Jul 25, 2008, at 9:46 AM, Duane Hill wrote:
>
> > For a test, ever thought about changing to a different script?
> >
> > Even through we have two Postfix border servers doing filtering, I
> > still have something on our internal (antiquated)
> CommuniGate server.
> >
> > I am using SA 3.2.5 with cgpsa and there hasn't been one bit of an
> > issue.
> >
> > http://www.tffenterprises.com/cgpsa/
> >
> > -d
>
>

**********************************************************************
Confidentiality : This e-mail and any attachments are intended for the
addressee only and may be confidential. If they come to you in error
you must take no action based on them, nor must you copy or show them
to anyone. Please advise the sender by replying to this e-mail
immediately and then delete the original from your computer.
Opinion : Any opinions expressed in this e-mail are entirely those of
the author and unless specifically stated to the contrary, are not
necessarily those of the author's employer.
Security Warning : Internet e-mail is not necessarily a secure
communications medium and can be subject to data corruption. We advise
that you consider this fact when e-mailing us.
Viruses : We have taken steps to ensure that this e-mail and any
attachments are free from known viruses but in keeping with good
computing practice, you should ensure that they are virus free.

Red Lion 49 Ltd T/A Solid State Logic
Registered as a limited company in England and Wales
(Company No:5362730)
Registered Office: 25 Spring Hill Road, Begbroke, Oxford OX5 1RU,
United Kingdom
**********************************************************************

[Robert - elists-2]

Ron

You are kinda shooting in the dark on the memory leak thread stuff.

You need to get a lil deeper in the system(s) to know.

I don't know what programming tools you are familiar with yet you could do
some tracing and know exactly what is going on and when it is happening and
then you can immediately correct or work on determining why and correct

 - rh