Correct Heap Space Settting for JAMES?

[Jerry M]

I'm getting a ton of Out of Memory errors in James in the connections
log. (James 2.3.1)  I've been running James for about 4 years and am
just now beginning to hit these Out Of Memory errors.  I don't see any
significant changes to the traffic patterns for mail, other than the
ever-growing spam count.  But spam shouldn't be causing OOMs, should
it?  This is not a high-traffic server, relatively speaking.  And there
are too many OOMs in the log for it to be related to one excessively
large email, etc.  The CPU hangs around 10% most of the time.

But regardless of whether I can identify why I'm getting the OOMs, I
have to fix this.  Where do you set the heap space allocation for
James?  I'm assuming the 'recommended' value is whatever the default is
set to.  But obviously that's not working for me.  So what is the
maximum safe value I can set the heap size to?

Thanks for a quick response to this so I can get my users back up and
running.

Jerry


---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@...
For additional commands, e-mail: server-user-help@...

[Josip Almasi]

Jerry M wrote:
> I'm getting a ton of Out of Memory errors in James in the connections
> log. (James 2.3.1)  I've been running James for about 4 years and am
> just now beginning to hit these Out Of Memory errors.  I don't see any
> significant changes to the traffic patterns for mail, other than the
> ever-growing spam count.  But spam shouldn't be causing OOMs, should
> it?  

Well it should not, but seems it does latelly (check spammers DoS my
James thread); I suggest you remove these spam msgs.

> But regardless of whether I can identify why I'm getting the OOMs, I
> have to fix this.  Where do you set the heap space allocation for
> James?  

In startup script.
But it won't help long term you know...

> I'm assuming the 'recommended' value is whatever the default is
> set to.  But obviously that's not working for me.  So what is the
> maximum safe value I can set the heap size to?

With 32-bit jre its somewhere between 1.6-1.8GB, with 64-bit much more.

Regards...

---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@...
For additional commands, e-mail: server-user-help@...

[Stefano Bagnara-2]

Josip Almasi ha scritto:
> Jerry M wrote:
>> I'm getting a ton of Out of Memory errors in James in the connections
>> log. (James 2.3.1)  I've been running James for about 4 years and am
>> just now beginning to hit these Out Of Memory errors.  I don't see any
>> significant changes to the traffic patterns for mail, other than the
>> ever-growing spam count.  But spam shouldn't be causing OOMs, should it?  
>
> Well it should not, but seems it does latelly (check spammers DoS my
> James thread); I suggest you remove these spam msgs.

James repository implementation try to keep in memory the keys list for
messages found in the repository.
Furthermore in case of file repository James will scan the folders at
startup before starting up the services.

I found myself emptying the "debug"/"spam" repositories very often to
avoid this issue.

Stefano

---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@...
For additional commands, e-mail: server-user-help@...

[Jerry M]

Thanks Stefano,

Are you saying that James stores in memory a key for every email found
for every user in the entire mail db?  Wow... I have to keep the 'spam'
messages around a while (several days at least) for my users to give
them time to check for messages incorrectly flagged as spam.  I only
have a couple of hundred users.  But with the current spam rate, my
inbox table has 175,000 records in it, even with pruning spam records
after a few days.  James is storing 175,000 keys in memory for this?  
I'm sure there's a reason for that.  But it does seem like it's destined
for hitting a memory heap limit sooner or later.  Is there any way to
turn that off?   How do JAMES installations with thousands of users
handle the memory problem?

Thanks.

Jerry

Jerry M wrote:

> I'm getting a ton of Out of Memory errors in James in the connections
> log. (James 2.3.1)  I've been running James for about 4 years and am
> just now beginning to hit these Out Of Memory errors.  I don't see any
> significant changes to the traffic patterns for mail, other than the
> ever-growing spam count.  But spam shouldn't be causing OOMs, should
> it?  This is not a high-traffic server, relatively speaking.  And
> there are too many OOMs in the log for it to be related to one
> excessively large email, etc.  The CPU hangs around 10% most of the time.
>
> But regardless of whether I can identify why I'm getting the OOMs, I
> have to fix this.  Where do you set the heap space allocation for
> James?  I'm assuming the 'recommended' value is whatever the default
> is set to.  But obviously that's not working for me.  So what is the
> maximum safe value I can set the heap size to?
>
> Thanks for a quick response to this so I can get my users back up and
> running.
>
> Jerry
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-user-unsubscribe@...
> For additional commands, e-mail: server-user-help@...
>
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@...
For additional commands, e-mail: server-user-help@...

[Jerry M]

Stephano,

I saw it about 20 times in a span of a few hours.  I don't know if it's
been happening in the past.  I just discovered it.  And I haven't seen
it since mid-day yesterday.

I'm a java programmer, and I understand stack dumps, etc.  But I'm not
sure what I need to do in order to capture a stack dump before the
problem occurs.  I'm willing to do whatever you suggest.  Just give me a
little more info on what to do.

Jerry


Stefano Bagnara wrote:

> Josip Almasi ha scritto:
>> Jerry M wrote:
>>> I'm getting a ton of Out of Memory errors in James in the
>>> connections log. (James 2.3.1)  I've been running James for about 4
>>> years and am just now beginning to hit these Out Of Memory errors.  
>>> I don't see any significant changes to the traffic patterns for
>>> mail, other than the ever-growing spam count.  But spam shouldn't be
>>> causing OOMs, should it?  
>>
>> Well it should not, but seems it does latelly (check spammers DoS my
>> James thread); I suggest you remove these spam msgs.
>
> James repository implementation try to keep in memory the keys list
> for messages found in the repository.
> Furthermore in case of file repository James will scan the folders at
> startup before starting up the services.
>
> I found myself emptying the "debug"/"spam" repositories very often to
> avoid this issue.
>
> Stefano
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-user-unsubscribe@...
> For additional commands, e-mail: server-user-help@...
>
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@...
For additional commands, e-mail: server-user-help@...

[Stefano Bagnara-2]

Jerry M ha scritto:

> Stephano,
>
> I saw it about 20 times in a span of a few hours.  I don't know if it's
> been happening in the past.  I just discovered it.  And I haven't seen
> it since mid-day yesterday.
>
> I'm a java programmer, and I understand stack dumps, etc.  But I'm not
> sure what I need to do in order to capture a stack dump before the
> problem occurs.  I'm willing to do whatever you suggest.  Just give me a
> little more info on what to do.

Unfortunately I'm lost on your error.
If you get the error again just try to get some stack dump.
You could also check the processlist for your mysql
(mysqladmin processlist) so to check whether there are multiple james
threads locked on the db that simply make new threads to time out.

(I often add a cron with mysqladmin processlist and the stack dumps and
log them so that when the error happens I can check the log around that
time).

Stefano

> Jerry
>
>
> Stefano Bagnara wrote:
>> Josip Almasi ha scritto:
>>> Jerry M wrote:
>>>> I'm getting a ton of Out of Memory errors in James in the
>>>> connections log. (James 2.3.1)  I've been running James for about 4
>>>> years and am just now beginning to hit these Out Of Memory errors.  
>>>> I don't see any significant changes to the traffic patterns for
>>>> mail, other than the ever-growing spam count.  But spam shouldn't be
>>>> causing OOMs, should it?  
>>>
>>> Well it should not, but seems it does latelly (check spammers DoS my
>>> James thread); I suggest you remove these spam msgs.
>>
>> James repository implementation try to keep in memory the keys list
>> for messages found in the repository.
>> Furthermore in case of file repository James will scan the folders at
>> startup before starting up the services.
>>
>> I found myself emptying the "debug"/"spam" repositories very often to
>> avoid this issue.
>>
>> Stefano
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-user-unsubscribe@...
>> For additional commands, e-mail: server-user-help@...
>>
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-user-unsubscribe@...
> For additional commands, e-mail: server-user-help@...
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@...
For additional commands, e-mail: server-user-help@...

[Stefano Bagnara-2]

Jerry M ha scritto:

> Thanks Stefano,
>
> Are you saying that James stores in memory a key for every email found
> for every user in the entire mail db?  Wow... I have to keep the 'spam'
> messages around a while (several days at least) for my users to give
> them time to check for messages incorrectly flagged as spam.  I only
> have a couple of hundred users.  But with the current spam rate, my
> inbox table has 175,000 records in it, even with pruning spam records
> after a few days.  James is storing 175,000 keys in memory for this?  
> I'm sure there's a reason for that.  But it does seem like it's destined
> for hitting a memory heap limit sooner or later.  Is there any way to
> turn that off?   How do JAMES installations with thousands of users
> handle the memory problem?

Unfortunately I only use JAMES in high traffic smtp server but with very
limited number of users (<100) so I never investigated issues with many
users.

The keys collection is loaded in memory when a "MailRepository.list"
method is called (for both db and file implementation). This is called
at least for POP3 connections.

For file based repositories, instead, the AvalonMailRepository
implementation manage a cache.
I know there is a way to disable the cache:
          <!-- in the file system. -->
          <repository
class="org.apache.james.mailrepository.AvalonMailRepository">
             <protocols>
                <protocol>file</protocol>
             </protocols>
             <types>
                <type>MAIL</type>
             </types>
             <!-- Set if the messages should be listed sorted. False by
default -->
             <config FIFO="false"/>
          </repository>

try to add a CACHEKEYS="false" to the config element.

Anyway the full key list will be loaded once in memory at initialize
time (and then disposed) and each time a list is called (and then
trashed by the garbage collector).
The cachekeys=false simply avoid having the full list permanently in memory.

Stefano


---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@...
For additional commands, e-mail: server-user-help@...

[Tor-Einar Jarnbjo-3]

Jerry M schrieb:
> I'm getting a ton of Out of Memory errors in James in the connections
> log. (James 2.3.1)  I've been running James for about 4 years and am
> just now beginning to hit these Out Of Memory errors.  I don't see any
> significant changes to the traffic patterns for mail, other than the
> ever-growing spam count.  But spam shouldn't be causing OOMs, should
> it?  This is not a high-traffic server, relatively speaking.  And
> there are too many OOMs in the log for it to be related to one
> excessively large email, etc.  The CPU hangs around 10% most of the time.
I used to get OutOfMemoryErrors from James as well, even if I don't have
very much traffic on my server. It turned out to be caused by large
mails, which are kept in memory by James during processing. I also have
some proprietary mailets configured, which might not be very optimized
when it comes to memory usage, so my server already puked when it got
mails reaching some 15-20MB in size.

The default heap size depends on which Java version you're using, but
you can increase it to e.g. 512MB by setting the following option at the
beginning of bin/phoenix.sh (or with setenv in phoenix.bat if you're
using Windows):

PHOENIX_JVM_OPTS=-Xmx512M

Tor



---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@...
For additional commands, e-mail: server-user-help@...

[Jerry M]

Tor,

Thanks, but several questions...

1) how did you determine that it was large emails causing the problem?

2) What did you change to fix it?

3) There is no phoenix.bat in my install.  I have a run.bat which looks
like what you are referring to.  I changed the heap size there, but
really didn't see any change.  So I'm wondering how run.bat works with
wrapper.  If I run James as a service, do I need to install the service
and reinstall the service to pick up the new parms?

4) I analyze OutOfMemory errors often in my day job.  But I'm curious
why there is no heap dump generated with each OOM?

5) I'm also trying to turn on verbose garbage collection, but nothing
seems to work to get it to generate vgc info.

Jerry


Tor-Einar Jarnbjo wrote:

> Jerry M schrieb:
>> I'm getting a ton of Out of Memory errors in James in the connections
>> log. (James 2.3.1)  I've been running James for about 4 years and am
>> just now beginning to hit these Out Of Memory errors.  I don't see
>> any significant changes to the traffic patterns for mail, other than
>> the ever-growing spam count.  But spam shouldn't be causing OOMs,
>> should it?  This is not a high-traffic server, relatively speaking.  
>> And there are too many OOMs in the log for it to be related to one
>> excessively large email, etc.  The CPU hangs around 10% most of the
>> time.
> I used to get OutOfMemoryErrors from James as well, even if I don't
> have very much traffic on my server. It turned out to be caused by
> large mails, which are kept in memory by James during processing. I
> also have some proprietary mailets configured, which might not be very
> optimized when it comes to memory usage, so my server already puked
> when it got mails reaching some 15-20MB in size.
>
> The default heap size depends on which Java version you're using, but
> you can increase it to e.g. 512MB by setting the following option at
> the beginning of bin/phoenix.sh (or with setenv in phoenix.bat if
> you're using Windows):
>
> PHOENIX_JVM_OPTS=-Xmx512M
>
> Tor
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-user-unsubscribe@...
> For additional commands, e-mail: server-user-help@...
>
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@...
For additional commands, e-mail: server-user-help@...