CPU usage quirk with multiple tomcat instances

[Dan Keeley-2]

Hi,

We have a Tomcat app that is loadbalanced by hardware.  I have 4 boxes
available in our test env, all of which vary slightly but not massively in
spec.

Anyway I moved from 2 tomcat instances on each machine ( listening on
different ports ) down to 1 and I noticed for the same load the cpu
utilisation dropped by 20%.  Interesting, how can this be explained?  Is
this the extra effort in tomcat or something?  Or is it memory related? the
2 instances had 1gb each, and the single one has been given 4gb.  However I
have jprofiled the instance and it has never seemed memory hungry.  Anyway I
suspect even at 2gb for 1 instance it would still be 20% less cpu - i can
always do a run to prove this if people think that necessary..

Another thing i've found is that on one server I have abnormally high CPU
utilisation for the same load.  I've checked the logging levels are all the
same as are the java VM and the VM config settings.  I wonder what else
could cause this?  All i can think is some sort of OS or hardware
malfunction! maybe the CPU cache is duff or something.

Many thanks to any input to my Friday weird-ness!
Dan

________________________________________________
Message sent using UebiMiau 2.7.10



---------------------------------------------------------------------
To start a new topic, e-mail: users@...
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[David Kerber]

Lists wrote:

> Hi,
>
> We have a Tomcat app that is loadbalanced by hardware.  I have 4 boxes
> available in our test env, all of which vary slightly but not massively in
> spec.
>
> Anyway I moved from 2 tomcat instances on each machine ( listening on
> different ports ) down to 1 and I noticed for the same load the cpu
> utilisation dropped by 20%.  Interesting, how can this be explained?  Is
>  

My guess (and that's all it is) is that you have eliminated much of the
context-switching overhead that two instances would have compared to one.

> this the extra effort in tomcat or something?  Or is it memory related? the
> 2 instances had 1gb each, and the single one has been given 4gb.  However I
> have jprofiled the instance and it has never seemed memory hungry.  Anyway I
> suspect even at 2gb for 1 instance it would still be 20% less cpu - i can
> always do a run to prove this if people think that necessary..
>
> Another thing i've found is that on one server I have abnormally high CPU
> utilisation for the same load.  I've checked the logging levels are all the
> same as are the java VM and the VM config settings.  I wonder what else
> could cause this?  All i can think is some sort of OS or hardware
> malfunction! maybe the CPU cache is duff or something.
>  

What is this cpu (type and speed) compared to the others?  If it's a
slower or less-capable (no numeric coprocessor, etc) cpu, the same load
would be a high percentage of its capability.

> Many thanks to any input to my Friday weird-ness!
> Dan
>  



---------------------------------------------------------------------
To start a new topic, e-mail: users@...
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Dan Keeley-2]

> My guess (and that's all it is) is that you have eliminated much of the
> context-switching overhead that two instances would have compared to one.

Ah I see.  I guess i can look at that with mpstat and look at things like
thread migrations etc.  Important because if we install more CPU's i guess
that may mitigate the context switching overhead.  Or maybe we can gain
performance by tying zones to one or two cpu's.
 
> What is this cpu (type and speed) compared to the others?  If it's a
> slower or less-capable (no numeric coprocessor, etc) cpu, the same load
> would be a high percentage of its capability.

Opteron 254 vs 252.  So 0.2ghz less but afaik no other difference.  Not
enough of a difference to account for the additional cpu.  so this box uses
80% cpu vs 30% on the other one for the same load.  the only other diff in
the box is that the better one is a Sun M2 X4100 rather than just a plain
X4100 - the only difference between those seem to be DDR2.
 

> > Many thanks to any input to my Friday weird-ness!
> > Dan
> >  
>
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@...
> To unsubscribe, e-mail: users-unsubscribe@...
> For additional commands, e-mail: users-help@...
>
>
>

________________________________________________
Message sent using UebiMiau 2.7.10



---------------------------------------------------------------------
To start a new topic, e-mail: users@...
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Christopher Schultz-2]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Da,

Lists wrote:
|> My guess (and that's all it is) is that you have eliminated much of the
|> context-switching overhead that two instances would have compared to one.
|
| Ah I see.  I guess i can look at that with mpstat and look at things like
| thread migrations etc.  Important because if we install more CPU's i guess
| that may mitigate the context switching overhead.  Or maybe we can gain
| performance by tying zones to one or two cpu's.

Is there any advantage to running multiple JVMs on a single machine? Why
not just give your single JVM a big chunk of memory and let it do its thing?

| Opteron 254 vs 252.  So 0.2ghz less but afaik no other difference.  Not
| enough of a difference to account for the additional cpu.  so this box
uses
| 80% cpu vs 30% on the other one for the same load.

Do you mean "same webapp load"? Obviously, that addition load is coming
from /somewhere/. What about other services running on the same machine?

- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkgdrqkACgkQ9CaO5/Lv0PCgqACfRxKediPbJyxkYpcBA0z1Qslt
5M8An1t6VYvyOh3+tMThcXqIZcHQFGVc
=y2AR
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To start a new topic, e-mail: users@...
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Dan Keeley-2]

Yes, same webapp load - i.e. they're both handling the same number of
transactions per second.

I guess the only advantage of multiple JVM's is reliability - if one
instance goes down the other can still handle load on that machine.

Also; I guess someone thought originally that 2 instances doing less work
would be easier for the VM to handle and therefore more performant than one
instance doing it all - clearly not!  ( I didnt set this architecture up -
If it's best practice to use one VM on each app server then fair enough -
i'd like to read up why tho! )

Thanks!
Dan

--------- Original Message --------
From: Tomcat Users List <users@...>
To: Tomcat Users List <users@...>
Subject: Re: CPU usage quirk with multiple tomcat instances
Date: 04/05/08 12:49

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Da,
>
> Lists wrote:
> |> My guess (and that's all it is) is that you have eliminated much of
the
> |> context-switching overhead that two instances would have compared to
one.
> |
> | Ah I see.  I guess i can look at that with mpstat and look at things
like
> | thread migrations etc.  Important because if we install more CPU's i
guess
> | that may mitigate the context switching overhead.  Or maybe we can gain
> | performance by tying zones to one or two cpu's.
>
> Is there any advantage to running multiple JVMs on a single machine? Why
> not just give your single JVM a big chunk of memory and let it do its
thing?
>
> | Opteron 254 vs 252.  So 0.2ghz less but afaik no other difference.  Not
> | enough of a difference to account for the additional cpu.  so this box
> uses
> | 80% cpu vs 30% on the other one for the same load.
>
> Do you mean "same webapp load"? Obviously, that addition load is
coming

> from /somewhere/. What about other services running on the same machine?
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>
> iEYEARECAAYFAkgdrqkACgkQ9CaO5/Lv0PCgqACfRxKediPbJyxkYpcBA0z1Qslt
> 5M8An1t6VYvyOh3+tMThcXqIZcHQFGVc
> =y2AR
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@...
> To unsubscribe, e-mail: users-unsubscribe@...
> For additional commands, e-mail: users-help@...
>
>
>

________________________________________________
Message sent using UebiMiau 2.7.10



---------------------------------------------------------------------
To start a new topic, e-mail: users@...
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Leon Rosenberg-3]

Hello Dan,

what also may happening is that you have unneeded or errorneous
synchronisation somewhere in your code. Two instances : less code in
same block and vm, effectively less synchronization.

try to profile whether some use cases take longer on the single tomcat
per machine.

We once tried to run multiple tomcats on one machine and the results
where opposed to yours: less performance, so i think your app behaves
"funny" :-)

regards
Leon


On Fri, May 2, 2008 at 2:42 PM, Lists <lists@...> wrote:

> Hi,
>
>  We have a Tomcat app that is loadbalanced by hardware.  I have 4 boxes
>  available in our test env, all of which vary slightly but not massively in
>  spec.
>
>  Anyway I moved from 2 tomcat instances on each machine ( listening on
>  different ports ) down to 1 and I noticed for the same load the cpu
>  utilisation dropped by 20%.  Interesting, how can this be explained?  Is
>  this the extra effort in tomcat or something?  Or is it memory related? the
>  2 instances had 1gb each, and the single one has been given 4gb.  However I
>  have jprofiled the instance and it has never seemed memory hungry.  Anyway I
>  suspect even at 2gb for 1 instance it would still be 20% less cpu - i can
>  always do a run to prove this if people think that necessary..
>
>  Another thing i've found is that on one server I have abnormally high CPU
>  utilisation for the same load.  I've checked the logging levels are all the
>  same as are the java VM and the VM config settings.  I wonder what else
>  could cause this?  All i can think is some sort of OS or hardware
>  malfunction! maybe the CPU cache is duff or something.
>
>  Many thanks to any input to my Friday weird-ness!
>  Dan
>
>  ________________________________________________
>  Message sent using UebiMiau 2.7.10
>
>
>
>  ---------------------------------------------------------------------
>  To start a new topic, e-mail: users@...
>  To unsubscribe, e-mail: users-unsubscribe@...
>  For additional commands, e-mail: users-help@...
>
>

---------------------------------------------------------------------
To start a new topic, e-mail: users@...
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Dan Keeley-2]

We dont have synchronisation issues - I've already profiled extensively with
jprofiler..  Well we did, but after using jprofiler we dont :)

Err, I think our app is the same as yours :) What im saying is that for the
same load the machine uses more cpu with 2 tomcat instances...  So i guess
that is correct, i was just looking for why!

Dan
----- Original Message -----
From: "Leon Rosenberg" <rosenberg.leon@...>
To: "Tomcat Users List" <users@...>; "Lists"
<lists@...>
Sent: Tuesday, May 06, 2008 9:33 AM
Subject: Re: CPU usage quirk with multiple tomcat instances

> Hello Dan,
>
> what also may happening is that you have unneeded or errorneous
> synchronisation somewhere in your code. Two instances : less code in
> same block and vm, effectively less synchronization.
>
> try to profile whether some use cases take longer on the single tomcat
> per machine.
>
> We once tried to run multiple tomcats on one machine and the results
> where opposed to yours: less performance, so i think your app behaves
> "funny" :-)
>
> regards
> Leon
>
>
> On Fri, May 2, 2008 at 2:42 PM, Lists <lists@...> wrote:
>> Hi,
>>
>>  We have a Tomcat app that is loadbalanced by hardware.  I have 4 boxes
>>  available in our test env, all of which vary slightly but not massively
>> in
>>  spec.
>>
>>  Anyway I moved from 2 tomcat instances on each machine ( listening on
>>  different ports ) down to 1 and I noticed for the same load the cpu
>>  utilisation dropped by 20%.  Interesting, how can this be explained?  Is
>>  this the extra effort in tomcat or something?  Or is it memory related?
>> the
>>  2 instances had 1gb each, and the single one has been given 4gb.
>> However I
>>  have jprofiled the instance and it has never seemed memory hungry.
>> Anyway I
>>  suspect even at 2gb for 1 instance it would still be 20% less cpu - i
>> can
>>  always do a run to prove this if people think that necessary..
>>
>>  Another thing i've found is that on one server I have abnormally high
>> CPU
>>  utilisation for the same load.  I've checked the logging levels are all
>> the
>>  same as are the java VM and the VM config settings.  I wonder what else
>>  could cause this?  All i can think is some sort of OS or hardware
>>  malfunction! maybe the CPU cache is duff or something.
>>
>>  Many thanks to any input to my Friday weird-ness!
>>  Dan
>>
>>  ________________________________________________
>>  Message sent using UebiMiau 2.7.10
>>
>>
>>
>>  ---------------------------------------------------------------------
>>  To start a new topic, e-mail: users@...
>>  To unsubscribe, e-mail: users-unsubscribe@...
>>  For additional commands, e-mail: users-help@...
>>
>>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@...
> To unsubscribe, e-mail: users-unsubscribe@...
> For additional commands, e-mail: users-help@...
>
>
>
>
> --
> No virus found in this incoming message.
> Checked by AVG.
> Version: 7.5.524 / Virus Database: 269.23.9/1418 - Release Date: 5/6/2008
> 5:17 PM
>
>

---------------------------------------------------------------------
To start a new topic, e-mail: users@...
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Leon Rosenberg-3]

btw, how do you measure it? vmstats? top?

On Wed, May 7, 2008 at 9:40 AM, Dan Keeley <lists@...> wrote:
> We dont have synchronisation issues - I've already profiled extensively with
> jprofiler..  Well we did, but after using jprofiler we dont :)
>
>  Err, I think our app is the same as yours :) What im saying is that for the
> same load the machine uses more cpu with 2 tomcat instances...  So i guess
> that is correct, i was just looking for why!
>
>  Dan

---------------------------------------------------------------------
To start a new topic, e-mail: users@...
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Dan Keeley-2]

we use sar and look at cpu utilisation..

--------- Original Message --------
From: Tomcat Users List <users@...>
To: Tomcat Users List <users@...>
Subject: Re: CPU usage quirk with multiple tomcat instances
Date: 07/05/08 08:13

> btw, how do you measure it? vmstats? top?
>
> On Wed, May 7, 2008 at 9:40 AM, Dan Keeley <lists@...>
wrote:
> > We dont have synchronisation issues - I've already profiled
extensively with
> > jprofiler..  Well we did, but after using jprofiler we dont :)
> >
> >  Err, I think our app is the same as yours :) What im saying is that
for the
> > same load the machine uses more cpu with 2 tomcat instances...  So i
guess

> > that is correct, i was just looking for why!
> >
> >  Dan
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@...
> To unsubscribe, e-mail: users-unsubscribe@...
> For additional commands, e-mail: users-help@...
>
>
>

________________________________________________
Message sent using UebiMiau 2.7.10



---------------------------------------------------------------------
To start a new topic, e-mail: users@...
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Leon Rosenberg-3]

but sar only shows you 10 minute averages ,right?
It would never show you any peaks.

leon

On Thu, May 8, 2008 at 12:09 PM, Lists <lists@...> wrote:

> we use sar and look at cpu utilisation..
>
> --------- Original Message --------
> From: Tomcat Users List <users@...>
> To: Tomcat Users List <users@...>
> Subject: Re: CPU usage quirk with multiple tomcat instances
> Date: 07/05/08 08:13
>
>> btw, how do you measure it? vmstats? top?
>>
>> On Wed, May 7, 2008 at 9:40 AM, Dan Keeley <lists@...>
> wrote:
>> > We dont have synchronisation issues - I've already profiled
> extensively with
>> > jprofiler..  Well we did, but after using jprofiler we dont :)
>> >
>> >  Err, I think our app is the same as yours :) What im saying is that
> for the
>> > same load the machine uses more cpu with 2 tomcat instances...  So i
> guess
>> > that is correct, i was just looking for why!
>> >
>> >  Dan
>>
>> ---------------------------------------------------------------------
>> To start a new topic, e-mail: users@...
>> To unsubscribe, e-mail: users-unsubscribe@...
>> For additional commands, e-mail: users-help@...
>>
>>
>>
>
> ________________________________________________
> Message sent using UebiMiau 2.7.10
>
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@...
> To unsubscribe, e-mail: users-unsubscribe@...
> For additional commands, e-mail: users-help@...
>
>

---------------------------------------------------------------------
To start a new topic, e-mail: users@...
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Dan Keeley-2]

no; We run sar and capture data every 5 seconds.

--------- Original Message --------
From: Tomcat Users List <users@...>
To: Tomcat Users List <users@...>, Lists
<lists@...>
Subject: Re: CPU usage quirk with multiple tomcat instances
Date: 08/05/08 10:21

> but sar only shows you 10 minute averages ,right?
> It would never show you any peaks.
>
> leon
>
> On Thu, May 8, 2008 at 12:09 PM, Lists <lists@...>
wrote:

> > we use sar and look at cpu utilisation..
> >
> > --------- Original Message --------
> > From: Tomcat Users List <users@...>
> > To: Tomcat Users List <users@...>
> > Subject: Re: CPU usage quirk with multiple tomcat instances
> > Date: 07/05/08 08:13
> >
> >> btw, how do you measure it? vmstats? top?
> >>
> >> On Wed, May 7, 2008 at 9:40 AM, Dan Keeley

&lt;lists@...&gt;
> > wrote:
> >> &gt; We dont have synchronisation issues - I've already
profiled
> > extensively with
> >> &gt; jprofiler..  Well we did, but after using jprofiler we
dont :)
> >> &gt;
> >> &gt;  Err, I think our app is the same as yours :) What im
saying is that
> > for the
> >> &gt; same load the machine uses more cpu with 2 tomcat
instances...  So i
> > guess
> >> &gt; that is correct, i was just looking for why!
> >> &gt;
> >> &gt;  Dan
> >>
> >>
---------------------------------------------------------------------

> >> To start a new topic, e-mail: users@...
> >> To unsubscribe, e-mail: users-unsubscribe@...
> >> For additional commands, e-mail: users-help@...
> >>
> >>
> >>
> >
> > ________________________________________________
> > Message sent using UebiMiau 2.7.10
> >
> >
> >
> > ---------------------------------------------------------------------
> > To start a new topic, e-mail: users@...
> > To unsubscribe, e-mail: users-unsubscribe@...
> > For additional commands, e-mail: users-help@...
> >
> >
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@...
> To unsubscribe, e-mail: users-unsubscribe@...
> For additional commands, e-mail: users-help@...
>
>
>

________________________________________________
Message sent using UebiMiau 2.7.10



---------------------------------------------------------------------
To start a new topic, e-mail: users@...
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...