Memory usage - what are we doing wrong?
|
View:
New views
8 Messages
—
Rating Filter:
Alert me
|
 |
Memory usage - what are we doing wrong?
by Marc Palmer Local
::
Rate this Message:
Hello all,
Ho hum I say. It's been a painful deployment today for many reasons. We have a 2GB Linux server running apache, mysql and a single tomcat instance. We have got two Grails apps running in the Tomcat VM... and we keep running out of memory and getting blank responses as the VM cannot allocate any native threads (native heap - not java heap). We're running with: export CATALINA_OPTS="-XX:MaxPermSize=300M -server -Xmx1000M" We tried running with more -Xmx but this just blows the heap quicker, and googling around shows that less java heap benefits your native heap, and we're running out of native threads/stack. Now our sites are hardly complex: +----------------------+-------+-------+ | Name | Files | LOC | +----------------------+-------+-------+ | Controllers | 8 | 346 | | Domain Classes | 7 | 103 | | Services | 3 | 178 | | Tag Libraries | 6 | 1149 | | Groovy Helpers | 3 | 135 | | Integration Tests | 2 | 8 | +----------------------+-------+-------+ | Totals | 29 | 1919 | +----------------------+-------+-------+ +----------------------+-------+-------+ | Name | Files | LOC | +----------------------+-------+-------+ | Controllers | 25 | 1504 | | Domain Classes | 25 | 438 | | Jobs | 2 | 24 | | Services | 10 | 565 | | Tag Libraries | 6 | 1015 | | Groovy Helpers | 5 | 131 | | Integration Tests | 20 | 104 | +----------------------+-------+-------+ | Totals | 93 | 3781 | +----------------------+-------+-------+ Now we must be able to get these to coexist with no trouble at all within 2GB ... or you'd have to say that there are serious issues scaling such trivial apps! What is interesting is that a lot of the trouble seems to be for some minutes after startup. Many pages come back blank and the log shows the error: SEVERE: Caught exception (java.lang.OutOfMemoryError: unable to create new native thread) executing org.apache.jk.common.ChannelSocket $SocketAcceptor@1f546b0, terminating thread ...but later reloads of those pages work, and seem to continue to work. It is newly loaded pages that present the problem - they often stick and don't respond (== native thread error in stack) but then a stop and reload works and there's never a problem on that page again. Compiling the GSPs is cpu intensive of course, but i see no reason why a modern computer will struggle with close to zero load just with one developer browsing around the sites. Nobody else is accessing these sites at midnight after lots of downtime (ugh the stories I could tell of today...) Anyway, input and experiences appreciated. It makes for very unimpressive startups... Marc --------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email |
|
|
Re: Memory usage - what are we doing wrong?
by Marc Palmer Local
::
Rate this Message:
On 29 Apr 2008, at 00:16, Marc Palmer wrote: > Hello all, > > Ho hum I say. It's been a painful deployment today for many reasons. > > We have a 2GB Linux server running apache, mysql and a single tomcat > instance. We have got two Grails apps running in the Tomcat VM... > and we keep running out of memory and getting blank responses as the > VM cannot allocate any native threads (native heap - not java heap). > We're running with: A lot of our native thread errors are as a result of opening db connections to local host: [1533789] [/].[default] Servlet.service() for servlet default threw exception java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:574) at java.util.Timer.<init>(Timer.java:154) at java.util.Timer.<init>(Timer.java:122) at com.mysql.jdbc.Connection.<init>(Connection.java:1441) at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java: 266) at java.sql.DriverManager.getConnection(DriverManager.java:525) at java.sql.DriverManager.getConnection(DriverManager.java:140) at org .springframework .jdbc .datasource .DriverManagerDataSource .getConnectionFromDriverManager(DriverManagerDataSource.java:283) at org .springframework .jdbc .datasource .DriverManagerDataSource .getConnectionFromDriverManager(DriverManagerDataSource.java:271) at org .springframework .jdbc .datasource .DriverManagerDataSource .getConnectionFromDriverManager(DriverManagerDataSource.java:255) at org .springframework .jdbc .datasource .DriverManagerDataSource.getConnection(DriverManagerDataSource.java:236) at org .springframework .orm .hibernate3 .LocalDataSourceConnectionProvider .getConnection(LocalDataSourceConnectionProvider.java:82) at org .hibernate .jdbc.ConnectionManager.openConnection(ConnectionManager.java:423) at org .hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java: 144) at org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:119) at org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:57) at org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java: 1326) at org .springframework .orm .hibernate3 .HibernateTransactionManager.doBegin(HibernateTransactionManager.java: 509) at org .springframework .transaction .support .AbstractPlatformTransactionManager .getTransaction(AbstractPlatformTransactionManager.java:377) Could it be that through caching / pooling we are leaking threads? Marc --------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email |
|
|
Re: Memory usage - what are we doing wrong?
by Mingfai
::
Rate this Message:
I'm obviously not an expert in Grails and I just speak from tuning a Java application.
from your message it sounds like there are problem with socket and connection. if it's about creating too much socket that consumes the memory, you may check at the OS level how many connections are made to the DB, e.g. netstat -an | grep 3306 and probably you want to review your apache and mod_jk and tomcat connector configuration to ensure it doesn't allow too much connections/open too many threads. There is also a general thing I will check to ensure reverse DNS lookup is disabled / the IP can be reverse looked up. If there are memory problem for just one user, have you ever done any profiling to see what objects consume the memory? and also enable GC logging/monitor GC activities. For "-XX:MaxPermSize=300M", I think it's useless for your case. Perm gen is for load things like class and methods and we usually need to increase it in development environment as our app server doesn't unload the class properly. Usually I'll use concurrent GC to minimize pause time but these kind of configuration shouldn't help your OutOfMemory problem anyway. For serious VM tuning, I'll review every VM parameters and tuning guide *[1] and monitor the GC chart and all performance indicators under a load test setup. Just my 2 cents Regards, mingfai [1] Reference http://java.sun.com/docs/hotspot/gc5.0/gc_tuning_5.html http://blogs.sun.com/watt/resource/jvm-options-list.html On Tue, Apr 29, 2008 at 7:26 AM, Marc Palmer <marc@...> wrote:
|
|
|
Re: Memory usage - what are we doing wrong?
by pratik patel-6
::
Rate this Message:
Check this page http://goobsoft.homeip.net/Wiki.jsp?page=JavaDebianTuning
You need to get a thread dump. On Solaris and JDK5 a "kill -QUIT PID" does this. This will show you the number of threads currently in use - though this can show a small number of threads in which case the solution may be elsewhere. Also, what happens when you run this on another server (a good test is to run this on another platform like Windo$e or Solaris). I haven't had any problems deploying a Grails app on either RH linux or Windoze - but my app is small with few users - though on par with the stats you posted for you app. |
|
|
Re: Memory usage - what are we doing wrong?
by Marc Pompl
::
Rate this Message:
Hi,
we have observed the same phenomena in our application from time to time, too. The memory consumption is low in general. Only the perm gen space is affected. I don't think, it is a problem with db connections due to the fact we are using connection pooling---for this there should be a limit in memory consumption. ;) In our application the phenoma is visible in one of the following behaviors: -- static content is "missing" and can not be requested by the browser -- a page cannot be visited---all others are available. Obviously, the application tells "access denied" but the access restrictions are all the same. May be, it is a problem in a plugin? Or is it created by a combination of plugins? In production I am using: -- cacheable -- jsecurity -- webtest (shouldn't be important, because it isn't "used") Best regards Marc Pompl Marc Palmer schrieb: > > On 29 Apr 2008, at 00:16, Marc Palmer wrote: >> Hello all, >> >> Ho hum I say. It's been a painful deployment today for many reasons. >> >> We have a 2GB Linux server running apache, mysql and a single tomcat >> instance. We have got two Grails apps running in the Tomcat VM... and >> we keep running out of memory and getting blank responses as the VM >> cannot allocate any native threads (native heap - not java heap). >> We're running with: > > A lot of our native thread errors are as a result of opening db > connections to local host: > > [1533789] [/].[default] Servlet.service() for servlet default threw > exception > java.lang.OutOfMemoryError: unable to create new native thread > at java.lang.Thread.start0(Native Method) > at java.lang.Thread.start(Thread.java:574) > at java.util.Timer.<init>(Timer.java:154) > at java.util.Timer.<init>(Timer.java:122) > at com.mysql.jdbc.Connection.<init>(Connection.java:1441) > at > com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:266) > > at java.sql.DriverManager.getConnection(DriverManager.java:525) > at java.sql.DriverManager.getConnection(DriverManager.java:140) > at > org.springframework.jdbc.datasource.DriverManagerDataSource.getConnectionFromDriverManager(DriverManagerDataSource.java:283) > > at > org.springframework.jdbc.datasource.DriverManagerDataSource.getConnectionFromDriverManager(DriverManagerDataSource.java:271) > > at > org.springframework.jdbc.datasource.DriverManagerDataSource.getConnectionFromDriverManager(DriverManagerDataSource.java:255) > > at > org.springframework.jdbc.datasource.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:236) > > at > org.springframework.orm.hibernate3.LocalDataSourceConnectionProvider.getConnection(LocalDataSourceConnectionProvider.java:82) > > at > org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:423) > > at > org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:144) > > at org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:119) > at > org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:57) > at > org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1326) > at > org.springframework.orm.hibernate3.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:509) > > at > org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:377) > > > > Could it be that through caching / pooling we are leaking threads? > > Marc > > > --------------------------------------------------------------------- > To unsubscribe from this list, please visit: > > http://xircles.codehaus.org/manage_email > > > --------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email |
|
|
|
|
|
Re: Memory usage - what are we doing wrong?
by justinedelson
::
Rate this Message:
Marc-
This stack trace suggests that you are not using connection pooling. Is that intentional? Not to say this is the cause of your problem - connection pooling shouldn't be necessary for the usage scenario you're describing but it's worth fixing. Justin On Mon, Apr 28, 2008 at 7:26 PM, Marc Palmer <marc@...> wrote:
|
|
|
Re: Memory usage - what are we doing wrong?
by Darren Davison
::
Rate this Message:
On Tue, Apr 29, 2008 at 12:26:41AM +0100, Marc Palmer wrote:
> org > .springframework > .jdbc > .datasource > .DriverManagerDataSource Marc, assume you are aware of the caveats of using this data source.. http://static.springframework.org/spring/docs/2.5.x/api/org/springframework/jdbc/datasource/DriverManagerDataSource.html I *think* it requires a thread per connection, so it's a likely candidate for the issues you're reporting. I'd recommend implementing c3p0. -- Darren Davison Public Key: 0xE855B3EA |
signature.asc (196 bytes) 
| Free Forum Powered by Nabble | Forum Help |