Memory leak in Tomcat 5.5.16

[Tom Price-3]

Hi,

I have a Java application that exposes a web services (SOAP/XML)
interface using Apache Tomcat and Axis2 (see detailed background below).
When this interface is heavily utilized, the application gradually leaks
old space memory until it runs out.  I have analyzed the heap usage on a
system where this memory leak has occurred, and I see that the following
classes take up most of the memory:

Size    Count   Class description
-------------------------------------------------------
433059616       557692  char[]
292410472       124245  byte[]
17801560        87232   * ConstMethodKlass
16543576        295421  org.apache.tomcat.util.buf.MessageBytes
15600840        390021  org.apache.tomcat.util.buf.ByteChunk
13372920        334323  org.apache.tomcat.util.buf.CharChunk
11575656        482319  java.lang.String

Does anyone know what could trigger the above classes to be leaked in
Tomcat 5.5.16?  I have also tried upgrading to Tomcat 5.5.26, but the
problem still occurs.

This is very similar to a memory leak posted to this list in 2006, but I
can't find what the resolution was in that case:

http://readlist.com/lists/tomcat.apache.org/users/5/27073.html

I have also consulted http://wiki.apache.org/tomcat/OutOfMemory but
couldn't find the solution.

The problem only seems to occur when multiple concurrent requests are
being received - if I issue requests in a single threaded manner, the
server can handle many hundreds of thousands of requests over an
extended period without any apparent memory leakage.  But if multiple
requests are sent concurrently, the application leaks about 1MB per 100
requests.

Here are the details of the application:

OS: Solaris 10 (Intel)
JRE: 1.5.0_07, launched from native C code (not the Java executable)
Tomcat: 5.5.16, embedded within my Java application
Axis2: 1.3

Thanks in advance for any help.

Tom.

---------------------------------------------------------------------
To start a new topic, e-mail: users@...
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Christopher Schultz-2]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Tom,

Tom Price wrote:
| I have analyzed the heap usage on a
| system where this memory leak has occurred, and I see that the following
| classes take up most of the memory:

[snip]

| Does anyone know what could trigger the above classes to be leaked in
| Tomcat 5.5.16?  I have also tried upgrading to Tomcat 5.5.26, but the
| problem still occurs.

Not to dismiss your problem, but these kinds of things tend to be
problems with the application rather than problems with Tomcat. Are you
able to use a profiler that can tell you where the object allocations
are actually occurring?

It's not unusual to have a ton of byte[] objects, char[] objects, String
objects, and Class objects. I'm guessing that the MessageBytes,
ByteChunk, and CharChunk objects in your heap summary are all related to
the byte[], char[], and String objects you can see there as well.

Have you been able to compare the numbers of those objects after, say,
100 requests with the same object counts after, say, 10000 requests? It
/is/ possible that Tomcat is leaking memory per connection, but very
unlikely given that thousands of servers are humming along quite nicely
without such problems.

It's also possible that Axis itself is doing some type of caching or
actually leaking memory somehow, but I also doubt that it does.

The only really good way to figure out what's happening is to instrument
your JVM and then test the hell out of it. Use JMeter to generate load
(serial or concurrent) and observe the heap, etc. You might even try to
disable some of the SOAP methods you are testing to see if this leak
only occurs with certain calls, or combinations of calls.

Good luck,
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkfr32wACgkQ9CaO5/Lv0PB5hACfY4Q88EAqc6ecvgOXKPwS3xtA
ufMAn3tSIziYrq8sNcw2CzTnw08rOoqF
=EgBf
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To start a new topic, e-mail: users@...
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Filip Hanik - Dev Lists]

try to disable the cache in conf/catalina.properties, at the bottom
there is a setting you can set to false
tomcat.util.buf.StringCache.byte.enabled=true

if the problem still exists, use jmap to dump the heap, and analyse the
actual dependencies for mem usage

Filip

Tom Price wrote:

> Hi,
>
> I have a Java application that exposes a web services (SOAP/XML)
> interface using Apache Tomcat and Axis2 (see detailed background below).
> When this interface is heavily utilized, the application gradually leaks
> old space memory until it runs out.  I have analyzed the heap usage on a
> system where this memory leak has occurred, and I see that the following
> classes take up most of the memory:
>
> Size    Count   Class description
> -------------------------------------------------------
> 433059616       557692  char[]
> 292410472       124245  byte[]
> 17801560        87232   * ConstMethodKlass
> 16543576        295421  org.apache.tomcat.util.buf.MessageBytes
> 15600840        390021  org.apache.tomcat.util.buf.ByteChunk
> 13372920        334323  org.apache.tomcat.util.buf.CharChunk
> 11575656        482319  java.lang.String
>
> Does anyone know what could trigger the above classes to be leaked in
> Tomcat 5.5.16?  I have also tried upgrading to Tomcat 5.5.26, but the
> problem still occurs.
>
> This is very similar to a memory leak posted to this list in 2006, but I
> can't find what the resolution was in that case:
>
> http://readlist.com/lists/tomcat.apache.org/users/5/27073.html
>
> I have also consulted http://wiki.apache.org/tomcat/OutOfMemory but
> couldn't find the solution.
>
> The problem only seems to occur when multiple concurrent requests are
> being received - if I issue requests in a single threaded manner, the
> server can handle many hundreds of thousands of requests over an
> extended period without any apparent memory leakage.  But if multiple
> requests are sent concurrently, the application leaks about 1MB per 100
> requests.
>
> Here are the details of the application:
>
> OS: Solaris 10 (Intel)
> JRE: 1.5.0_07, launched from native C code (not the Java executable)
> Tomcat: 5.5.16, embedded within my Java application
> Axis2: 1.3
>
> Thanks in advance for any help.
>
> Tom.
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@...
> To unsubscribe, e-mail: users-unsubscribe@...
> For additional commands, e-mail: users-help@...
>
>
>
>  

---------------------------------------------------------------------
To start a new topic, e-mail: users@...
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Tom Price-3]

Christopher,

Christopher Schultz wrote:
> Have you been able to compare the numbers of those objects after, say,
> 100 requests with the same object counts after, say, 10000 requests?
It
> /is/ possible that Tomcat is leaking memory per connection, but very
> unlikely given that thousands of servers are humming along quite
nicely
> without such problems.

Thanks for the suggestions.  I have looked at the change in number of
objects over time, and the only objects that appear to be leaked
consistently are the following (the number is the increase in object
count over a typical 10 minute period, with about 2000 requests):

org.apache.tomcat.util.buf.ByteChunk 10437
org.apache.tomcat.util.buf.CharChunk 8946
org.apache.tomcat.util.buf.MessageBytes 7881

Other object types, including byte[] and char[], vary wildly in each 10
minute period, presumably depending on exactly what's going on at the
time of the sample.

I'm trying to find a tool that will show me which objects are holding
onto references to these Tomcat objects, but haven't had much success so
far - any suggestions gratefully received.

> The only really good way to figure out what's happening is to
instrument
> your JVM and then test the hell out of it. Use JMeter to generate load
> (serial or concurrent) and observe the heap, etc. You might even try
to
> disable some of the SOAP methods you are testing to see if this leak
> only occurs with certain calls, or combinations of calls.

That's what I'm doing now...  I will also try Filip's suggestion of
disabling string caching.  Sadly jmap doesn't work on this system
because the heap is large and jmap always crashes with an
OutOfMemoryError due to the 2GB heap limit in the JVM.  I haven't used
JMeter before but will look into it.

Thanks,
Tom.

---------------------------------------------------------------------
To start a new topic, e-mail: users@...
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[markt-2]

Tom Price
> I'm trying to find a tool that will show me which objects are holding
> onto references to these Tomcat objects, but haven't had much success so
> far - any suggestions gratefully received.

Not free, but I use YourKit and get on very well with it.

Mark


---------------------------------------------------------------------
To start a new topic, e-mail: users@...
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[hai_vu]

I use Netbeans 6 with its built-in profiler (all free) to track down
memory usage issues, and other performance problems in Java apps in
general and Tomcat in particular.

Hai Vu



"Tom Price" <Tom.Price@...>
27/03/2008 07:59 PM
Please respond to
"Tomcat Users List" <users@...>


To
"Tomcat Users List" <users@...>
cc

Subject
RE: Memory leak in Tomcat 5.5.16






Christopher,

Christopher Schultz wrote:
> Have you been able to compare the numbers of those objects after, say,
> 100 requests with the same object counts after, say, 10000 requests?
It
> /is/ possible that Tomcat is leaking memory per connection, but very
> unlikely given that thousands of servers are humming along quite
nicely
> without such problems.

Thanks for the suggestions.  I have looked at the change in number of
objects over time, and the only objects that appear to be leaked
consistently are the following (the number is the increase in object
count over a typical 10 minute period, with about 2000 requests):

org.apache.tomcat.util.buf.ByteChunk             10437
org.apache.tomcat.util.buf.CharChunk             8946
org.apache.tomcat.util.buf.MessageBytes          7881

Other object types, including byte[] and char[], vary wildly in each 10
minute period, presumably depending on exactly what's going on at the
time of the sample.

I'm trying to find a tool that will show me which objects are holding
onto references to these Tomcat objects, but haven't had much success so
far - any suggestions gratefully received.

> The only really good way to figure out what's happening is to
instrument
> your JVM and then test the hell out of it. Use JMeter to generate load
> (serial or concurrent) and observe the heap, etc. You might even try
to
> disable some of the SOAP methods you are testing to see if this leak
> only occurs with certain calls, or combinations of calls.

That's what I'm doing now...  I will also try Filip's suggestion of
disabling string caching.  Sadly jmap doesn't work on this system
because the heap is large and jmap always crashes with an
OutOfMemoryError due to the 2GB heap limit in the JVM.  I haven't used
JMeter before but will look into it.

Thanks,
Tom.

---------------------------------------------------------------------
To start a new topic, e-mail: users@...
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Tom Price-3]

Hi,

I have now managed to do some analysis of the classes that are
referencing the leaked objects, can anyone help me interpret these
results?

I got jmap/jhat working by upgrading to JRE 1.6.0_05, and took a heap
dump at a time after a period of stress when all SOAP/XML connections
had been closed (so no requests were outstanding).  I had also forced
full garbage collection a few times.  There were many
org.apache.tomcat.util.buf.MessageBytes objects still referenced, which
all seem to be held directly and indirectly by org.apache.coyote.Request
objects.  These are referenced by several classes, which is where I get
a bit lost:

org.apache.coyote.http11.InternalInputBuffer@0xb1b129a0 (70 bytes) :
field request
org.apache.coyote.http11.Http11Processor@0xb1e30148 (120 bytes) : field
request
org.apache.catalina.connector.Request@0xb2348310 (153 bytes) : field
coyoteRequest
org.apache.coyote.RequestInfo@0xb1b12ad8 (68 bytes) : field req
org.apache.catalina.connector.InputBuffer@0xb2ef90b8 (78 bytes) : field
coyoteRequest
org.apache.coyote.Response@0xb2767d48 (78 bytes) : field req

Where do I go next to find out who is keeping these Request objects
alive?

Thanks,
Tom.

---------------------------------------------------------------------
To start a new topic, e-mail: users@...
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Tom Price-3]

Hi all,

No more help required - I traced back all the references to the Request
objects and it did turn out to be a bug in my application code.  One of
my tracing classes (written a long time ago, before we used Tomcat) was
caching all created Thread objects.  So when Tomcat decided to allocate
new threads to process incoming requests (and un-referenced the old
ones), my application cached the old threads and they slowly leaked.
I'm assuming that Tomcat only does this when handling multiple
concurrent connections (i.e. threads are reused when handling
single-threaded connections), hence the fact that I could only reproduce
the problem when sending in concurrent requests.

Thanks for everyone's suggestions.

Tom.

---------------------------------------------------------------------
To start a new topic, e-mail: users@...
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...