Remote client performance

[David Blevins]

We've had more than one report of windows users running into  
performance issues with the way we manage sockets between the client  
and server.  Typically they run out of ports, etc.  A bit back Dain  
got The Grinder[1] running and showed me how good the performance was  
when we dropped in Jetty and ran our ejbd protocol over http.  On my  
machine with a simple ejb, 2 processes, 10 threads each (i.e. 20  
threads totoal), I was getting about 2500 TPS (transactions per  
second) over jetty+http+ejbd.  That was several times our raw ejbd  
performance which would start at around 600-700 TPS and taper off  
after a few hours to next to nothing.

I looked into how much of the Jetty code we'd need to run that setup  
all the time and it was around 240K so I decided to give improving our  
raw socket + ejbd setup another go and read up on NIO and all that  
business.  I added some keep-alive logic, buffering, eliminated some  
flushing, and introduced another thread to check for and close  
connections being kept alive in the "waiting for request" state too  
long (3 second cut off currently).

Here's the performance we're getting now:

   http://people.apache.org/~dblevins/ejbd-client-performance.png

The number to look at is the mean, which is 7300 TPS.  I kicked this  
off yesterday and it's been running pretty consistently at that number  
for about 21 hours now.  It actually starts at about 7200 TPS and goes  
up just slightly, perhaps because of vm hotspotting, though not too  
sure.

Anyway this should eliminate any performance issues someone might have  
seen before with heavily active remote clients.  It's not yet using  
NIO.  I'd like to try it out, but might keep that for next release.

There are still good performance related reasons to use the ejbd over  
http/jetty setup.  Here's the basic trade off:

  - ejbd over raw socket:
    - high TPS with lower number of active clients (hundred or so vms)
    - low TPS with larger number of active clients (more than a  
hundred or so)

  - ejbd over http/jetty:
    - low TPS with lower number of active clients (less than a hundred)
    - high TPS with larger number of active clients (hundreds or  
thousands)

Note that client doesn't mean threads, it means actual VMs.  You can  
have several client threads going with pretty much no impact.

-David

[1] http://grinder.sourceforge.net/

[manucet]

Pretty cool stuff Dave. Why is the performance better for high TPS for
ejbd over http/jetty? As the no of clients increases does the TPS
decrease for ejbd over raw socket?

Regards
Manu

On Mon, Jul 14, 2008 at 3:18 AM, David Blevins <david.blevins@...> wrote:

> We've had more than one report of windows users running into performance
> issues with the way we manage sockets between the client and server.
>  Typically they run out of ports, etc.  A bit back Dain got The Grinder[1]
> running and showed me how good the performance was when we dropped in Jetty
> and ran our ejbd protocol over http.  On my machine with a simple ejb, 2
> processes, 10 threads each (i.e. 20 threads totoal), I was getting about
> 2500 TPS (transactions per second) over jetty+http+ejbd.  That was several
> times our raw ejbd performance which would start at around 600-700 TPS and
> taper off after a few hours to next to nothing.
>
> I looked into how much of the Jetty code we'd need to run that setup all the
> time and it was around 240K so I decided to give improving our raw socket +
> ejbd setup another go and read up on NIO and all that business.  I added
> some keep-alive logic, buffering, eliminated some flushing, and introduced
> another thread to check for and close connections being kept alive in the
> "waiting for request" state too long (3 second cut off currently).
>
> Here's the performance we're getting now:
>
>  http://people.apache.org/~dblevins/ejbd-client-performance.png
>
> The number to look at is the mean, which is 7300 TPS.  I kicked this off
> yesterday and it's been running pretty consistently at that number for about
> 21 hours now.  It actually starts at about 7200 TPS and goes up just
> slightly, perhaps because of vm hotspotting, though not too sure.
>
> Anyway this should eliminate any performance issues someone might have seen
> before with heavily active remote clients.  It's not yet using NIO.  I'd
> like to try it out, but might keep that for next release.
>
> There are still good performance related reasons to use the ejbd over
> http/jetty setup.  Here's the basic trade off:
>
>  - ejbd over raw socket:
>   - high TPS with lower number of active clients (hundred or so vms)
>   - low TPS with larger number of active clients (more than a hundred or so)
>
>  - ejbd over http/jetty:
>   - low TPS with lower number of active clients (less than a hundred)
>   - high TPS with larger number of active clients (hundreds or thousands)
>
> Note that client doesn't mean threads, it means actual VMs.  You can have
> several client threads going with pretty much no impact.
>
> -David
>
> [1] http://grinder.sourceforge.net/
>

[KMalhi]

The TPS number is very impressive. If its not a complicated setup, could you
share some details on how I could replicate this test on my machine. If we
run it on our machines, then we can probably post the results of running on
different configurations (on our website)

On Sun, Jul 13, 2008 at 5:48 PM, David Blevins <david.blevins@...>
wrote:

> We've had more than one report of windows users running into performance
> issues with the way we manage sockets between the client and server.
>  Typically they run out of ports, etc.  A bit back Dain got The Grinder[1]
> running and showed me how good the performance was when we dropped in Jetty
> and ran our ejbd protocol over http.  On my machine with a simple ejb, 2
> processes, 10 threads each (i.e. 20 threads totoal), I was getting about
> 2500 TPS (transactions per second) over jetty+http+ejbd.  That was several
> times our raw ejbd performance which would start at around 600-700 TPS and
> taper off after a few hours to next to nothing.
>
> I looked into how much of the Jetty code we'd need to run that setup all
> the time and it was around 240K so I decided to give improving our raw
> socket + ejbd setup another go and read up on NIO and all that business.  I
> added some keep-alive logic, buffering, eliminated some flushing, and
> introduced another thread to check for and close connections being kept
> alive in the "waiting for request" state too long (3 second cut off
> currently).
>
> Here's the performance we're getting now:
>
>  http://people.apache.org/~dblevins/ejbd-client-performance.png<http://people.apache.org/%7Edblevins/ejbd-client-performance.png>
>
> The number to look at is the mean, which is 7300 TPS.  I kicked this off
> yesterday and it's been running pretty consistently at that number for about
> 21 hours now.  It actually starts at about 7200 TPS and goes up just
> slightly, perhaps because of vm hotspotting, though not too sure.
>
> Anyway this should eliminate any performance issues someone might have seen
> before with heavily active remote clients.  It's not yet using NIO.  I'd
> like to try it out, but might keep that for next release.
>
> There are still good performance related reasons to use the ejbd over
> http/jetty setup.  Here's the basic trade off:
>
>  - ejbd over raw socket:
>   - high TPS with lower number of active clients (hundred or so vms)
>   - low TPS with larger number of active clients (more than a hundred or
> so)
>
>  - ejbd over http/jetty:
>   - low TPS with lower number of active clients (less than a hundred)
>   - high TPS with larger number of active clients (hundreds or thousands)
>
> Note that client doesn't mean threads, it means actual VMs.  You can have
> several client threads going with pretty much no impact.
>
> -David
>
> [1] http://grinder.sourceforge.net/
>

--
Karan Singh Malhi

[David Blevins]

On Jul 13, 2008, at 3:03 PM, Manu George wrote:

> Pretty cool stuff Dave. Why is the performance better for high TPS for
> ejbd over http/jetty?

Don't know why we're 3x faster on the 2 client + 20 threads side of  
things.  It might simply be that we have an advantage in that we can  
completely control the client side.  I saw the biggest boost in  
performance when I made the client buffer up nearly all it's data  
before writing any of it.  I also made it so the client can have at  
most one open connection per server URL and all requests file through  
it.

For the record you can actually get more connections per client by  
slightly appending to your server url as so:

  ejbd://localhost:4201/a
  ejbd://localhost:4201/b

Everything after the "/" is currently ignored, but will allow you to  
get another connection to the same server if you think you really need  
it.  I actually saw pretty great performance with just one connection  
shared among 10 client threads, but maybe you want a second connection  
open for very long running requests as to not block up the shorter  
requests.


> As the no of clients increases does the TPS
> decrease for ejbd over raw socket?

We're still using regular java io so we have to dedicate a thread to  
each open connection.  So we're limited by the number of threads that  
can realistically be running without too much context switching.  On a  
dual core you can run quite a few, on quad core you can run an  
impressive number, on an Azul box ... sky is the limit practically.  
However if you do exceed your set thread pool size, new clients are  
simply not going to get a connection till someone else gives theirs up  
or it times out after 3 seconds of inactivity (the timeout will be  
configurable).  Works great if you have dozens clients (or more if you  
have more cores or CPUs) who just want to wail on the server  
continuously.

Jetty uses NIO and can have one thread dedicated to reading data from  
open connections (using NIO Selectors).   It'll buffer up data till  
there's enough for a whole request *then* it will send it into the  
thread pool for work.  So the number of clients you can have is not  
strictly tied to the number of threads.  So if you had a huge burst in  
client count and requests it could likely handle that just fine.  
Works great if the number of clients can fluctuate erratically from a  
few dozen to several thousand on a regular basis.


-David

> On Mon, Jul 14, 2008 at 3:18 AM, David Blevins  
> <david.blevins@...> wrote:
>> We've had more than one report of windows users running into  
>> performance
>> issues with the way we manage sockets between the client and server.
>> Typically they run out of ports, etc.  A bit back Dain got The  
>> Grinder[1]
>> running and showed me how good the performance was when we dropped  
>> in Jetty
>> and ran our ejbd protocol over http.  On my machine with a simple  
>> ejb, 2
>> processes, 10 threads each (i.e. 20 threads totoal), I was getting  
>> about
>> 2500 TPS (transactions per second) over jetty+http+ejbd.  That was  
>> several
>> times our raw ejbd performance which would start at around 600-700  
>> TPS and
>> taper off after a few hours to next to nothing.
>>
>> I looked into how much of the Jetty code we'd need to run that  
>> setup all the
>> time and it was around 240K so I decided to give improving our raw  
>> socket +
>> ejbd setup another go and read up on NIO and all that business.  I  
>> added
>> some keep-alive logic, buffering, eliminated some flushing, and  
>> introduced
>> another thread to check for and close connections being kept alive  
>> in the
>> "waiting for request" state too long (3 second cut off currently).
>>
>> Here's the performance we're getting now:
>>
>> http://people.apache.org/~dblevins/ejbd-client-performance.png
>>
>> The number to look at is the mean, which is 7300 TPS.  I kicked  
>> this off
>> yesterday and it's been running pretty consistently at that number  
>> for about
>> 21 hours now.  It actually starts at about 7200 TPS and goes up just
>> slightly, perhaps because of vm hotspotting, though not too sure.
>>
>> Anyway this should eliminate any performance issues someone might  
>> have seen
>> before with heavily active remote clients.  It's not yet using  
>> NIO.  I'd
>> like to try it out, but might keep that for next release.
>>
>> There are still good performance related reasons to use the ejbd over
>> http/jetty setup.  Here's the basic trade off:
>>
>> - ejbd over raw socket:
>>  - high TPS with lower number of active clients (hundred or so vms)
>>  - low TPS with larger number of active clients (more than a  
>> hundred or so)
>>
>> - ejbd over http/jetty:
>>  - low TPS with lower number of active clients (less than a hundred)
>>  - high TPS with larger number of active clients (hundreds or  
>> thousands)
>>
>> Note that client doesn't mean threads, it means actual VMs.  You  
>> can have
>> several client threads going with pretty much no impact.
>>
>> -David
>>
>> [1] http://grinder.sourceforge.net/
>>
>

[David Blevins]

On Jul 13, 2008, at 3:13 PM, Karan Malhi wrote:

> The TPS number is very impressive. If its not a complicated setup,  
> could you
> share some details on how I could replicate this test on my machine.  
> If we
> run it on our machines, then we can probably post the results of  
> running on
> different configurations (on our website)

Something like that would be pretty great.

http://people.apache.org/~dblevins/grinder.tar.gz

Zipped up my setup and uploaded it.  You just need to tweak the  
grinder.properties file.  I also hacked on the bin/openejb script a  
bit, though that's not required.  The grinder.py file controls the  
client code, though i don't fully understand it.

You just start things in this order:

   ./openejb-3.1-SNAPSHOT/bin/openejb start
   ./grinder-3.0.1/bin/console
   ./grinder-3.0.1/bin/agent

Then you click the go button on the left of the console.

-David

[Dain Sundstrom]

On Jul 13, 2008, at 6:39 PM, David Blevins wrote:

>
> On Jul 13, 2008, at 3:13 PM, Karan Malhi wrote:
>
>> The TPS number is very impressive. If its not a complicated setup,  
>> could you
>> share some details on how I could replicate this test on my  
>> machine. If we
>> run it on our machines, then we can probably post the results of  
>> running on
>> different configurations (on our website)
>
> Something like that would be pretty great.
>
> http://people.apache.org/~dblevins/grinder.tar.gz
>
> Zipped up my setup and uploaded it.  You just need to tweak the  
> grinder.properties file.  I also hacked on the bin/openejb script a  
> bit, though that's not required.  The grinder.py file controls the  
> client code, though i don't fully understand it.
>
> You just start things in this order:
>
>  ./openejb-3.1-SNAPSHOT/bin/openejb start
>  ./grinder-3.0.1/bin/console
>  ./grinder-3.0.1/bin/agent
>
> Then you click the go button on the left of the console.

Between each run you will want to restart the console.  There is a bug  
in Grinder where the calculated statistics get messed up when you run  
a second time in the console (is divides TPS by 10 for each restart).

-dain

[manucet]

Saw an interesting discussion regarding NIO/IO performance on linux.
If this is still valid then moving to NIO will not increase
performance on linux

http://www.theserverside.com/discussions/thread.tss?thread_id=26700

Regards
Manu
On Mon, Jul 14, 2008 at 10:53 PM, Dain Sundstrom <dain@...> wrote:

> On Jul 13, 2008, at 6:39 PM, David Blevins wrote:
>
>>
>> On Jul 13, 2008, at 3:13 PM, Karan Malhi wrote:
>>
>>> The TPS number is very impressive. If its not a complicated setup, could
>>> you
>>> share some details on how I could replicate this test on my machine. If
>>> we
>>> run it on our machines, then we can probably post the results of running
>>> on
>>> different configurations (on our website)
>>
>> Something like that would be pretty great.
>>
>> http://people.apache.org/~dblevins/grinder.tar.gz
>>
>> Zipped up my setup and uploaded it.  You just need to tweak the
>> grinder.properties file.  I also hacked on the bin/openejb script a bit,
>> though that's not required.  The grinder.py file controls the client code,
>> though i don't fully understand it.
>>
>> You just start things in this order:
>>
>>  ./openejb-3.1-SNAPSHOT/bin/openejb start
>>  ./grinder-3.0.1/bin/console
>>  ./grinder-3.0.1/bin/agent
>>
>> Then you click the go button on the left of the console.
>
> Between each run you will want to restart the console.  There is a bug in
> Grinder where the calculated statistics get messed up when you run a second
> time in the console (is divides TPS by 10 for each restart).
>
> -dain
>
>

[Dain Sundstrom]

The first 10+ revisions of NIO were terrible, but I have heard that it  
has gotten much better in the last couple of years.  Hopefully if  
David gets NIO working we can have a "nio=true" option in the ejbd  
server.

-dain

On Jul 14, 2008, at 11:23 AM, Manu George wrote:

> Saw an interesting discussion regarding NIO/IO performance on linux.
> If this is still valid then moving to NIO will not increase
> performance on linux
>
> http://www.theserverside.com/discussions/thread.tss?thread_id=26700
>
> Regards
> Manu
> On Mon, Jul 14, 2008 at 10:53 PM, Dain Sundstrom <dain@...>  
> wrote:
>> On Jul 13, 2008, at 6:39 PM, David Blevins wrote:
>>
>>>
>>> On Jul 13, 2008, at 3:13 PM, Karan Malhi wrote:
>>>
>>>> The TPS number is very impressive. If its not a complicated  
>>>> setup, could
>>>> you
>>>> share some details on how I could replicate this test on my  
>>>> machine. If
>>>> we
>>>> run it on our machines, then we can probably post the results of  
>>>> running
>>>> on
>>>> different configurations (on our website)
>>>
>>> Something like that would be pretty great.
>>>
>>> http://people.apache.org/~dblevins/grinder.tar.gz
>>>
>>> Zipped up my setup and uploaded it.  You just need to tweak the
>>> grinder.properties file.  I also hacked on the bin/openejb script  
>>> a bit,
>>> though that's not required.  The grinder.py file controls the  
>>> client code,
>>> though i don't fully understand it.
>>>
>>> You just start things in this order:
>>>
>>> ./openejb-3.1-SNAPSHOT/bin/openejb start
>>> ./grinder-3.0.1/bin/console
>>> ./grinder-3.0.1/bin/agent
>>>
>>> Then you click the go button on the left of the console.
>>
>> Between each run you will want to restart the console.  There is a  
>> bug in
>> Grinder where the calculated statistics get messed up when you run  
>> a second
>> time in the console (is divides TPS by 10 for each restart).
>>
>> -dain
>>
>>

[manucet]

Good to know that NIO is better now :). Would be interesting to see
the difference once David gets it working

Regards
Manu

On Mon, Jul 14, 2008 at 11:57 PM, Dain Sundstrom <dain@...> wrote:

> The first 10+ revisions of NIO were terrible, but I have heard that it has
> gotten much better in the last couple of years.  Hopefully if David gets NIO
> working we can have a "nio=true" option in the ejbd server.
>
> -dain
>
> On Jul 14, 2008, at 11:23 AM, Manu George wrote:
>
>> Saw an interesting discussion regarding NIO/IO performance on linux.
>> If this is still valid then moving to NIO will not increase
>> performance on linux
>>
>> http://www.theserverside.com/discussions/thread.tss?thread_id=26700
>>
>> Regards
>> Manu
>> On Mon, Jul 14, 2008 at 10:53 PM, Dain Sundstrom <dain@...> wrote:
>>>
>>> On Jul 13, 2008, at 6:39 PM, David Blevins wrote:
>>>
>>>>
>>>> On Jul 13, 2008, at 3:13 PM, Karan Malhi wrote:
>>>>
>>>>> The TPS number is very impressive. If its not a complicated setup,
>>>>> could
>>>>> you
>>>>> share some details on how I could replicate this test on my machine. If
>>>>> we
>>>>> run it on our machines, then we can probably post the results of
>>>>> running
>>>>> on
>>>>> different configurations (on our website)
>>>>
>>>> Something like that would be pretty great.
>>>>
>>>> http://people.apache.org/~dblevins/grinder.tar.gz
>>>>
>>>> Zipped up my setup and uploaded it.  You just need to tweak the
>>>> grinder.properties file.  I also hacked on the bin/openejb script a bit,
>>>> though that's not required.  The grinder.py file controls the client
>>>> code,
>>>> though i don't fully understand it.
>>>>
>>>> You just start things in this order:
>>>>
>>>> ./openejb-3.1-SNAPSHOT/bin/openejb start
>>>> ./grinder-3.0.1/bin/console
>>>> ./grinder-3.0.1/bin/agent
>>>>
>>>> Then you click the go button on the left of the console.
>>>
>>> Between each run you will want to restart the console.  There is a bug in
>>> Grinder where the calculated statistics get messed up when you run a
>>> second
>>> time in the console (is divides TPS by 10 for each restart).
>>>
>>> -dain
>>>
>>>
>
>