Keep-Alive timeout & persistent connections
|
View:
New views
15 Messages
—
Rating Filter:
Alert me
|
 |
Keep-Alive timeout & persistent connections
by Sam Berlin
::
Rate this Message:
Hi Folks,
We just stumbled across an issue with persistent connections where the server is sending a Connection: Keep-Alive header, but also a Keep-Alive header with a timeout=## (and a max=##). HttpClient's connection management sees that the connection can be kept alive, so pools it for potential reuse, but doesn't respect the timeout (or max) values. The problem is that the connection remains opens from the eyes of HttpClient. A subsequent request sends correctly (with no IOX's, oddly enough), but reading causes HttpClient to read a -1 and throw a NoHttpResponseException. This is against a run-of-the-mill Apache httpd server. This is normally supposed to be handled by a HttpRequestRetryHandler, to retry to the connection, but I'm wondering if it shouldn't even get there in the first place. Should HttpClient respect the Keep-Alive header's timeout (and max, or other) token and use a new connection in the first place? Sam --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscribe@... For additional commands, e-mail: dev-help@... |
|
|
Re: Keep-Alive timeout & persistent connections
by olegk
::
Rate this Message:
On Wed, 2008-06-25 at 17:14 -0400, Sam Berlin wrote:
> Hi Folks, > > We just stumbled across an issue with persistent connections where the > server is sending a Connection: Keep-Alive header, but also a > Keep-Alive header with a timeout=## (and a max=##). HttpClient's > connection management sees that the connection can be kept alive, so > pools it for potential reuse, but doesn't respect the timeout (or max) > values. The problem is that the connection remains opens from the > eyes of HttpClient. A subsequent request sends correctly (with no > IOX's, oddly enough), but reading causes HttpClient to read a -1 and > throw a NoHttpResponseException. This is against a run-of-the-mill > Apache httpd server. > > This is normally supposed to be handled by a HttpRequestRetryHandler, > to retry to the connection, but I'm wondering if it shouldn't even get > there in the first place. Should HttpClient respect the Keep-Alive > header's timeout (and max, or other) token and use a new connection in > the first place? > Hi Sam We certainly need to fix that. Could you please open a JIRA for this issue? Oleg > Sam > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscribe@... > For additional commands, e-mail: dev-help@... > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscribe@... For additional commands, e-mail: dev-help@... |
|
|
Re: Keep-Alive timeout & persistent connections
by Sam Berlin
::
Rate this Message:
Done @ https://issues.apache.org/jira/browse/HTTPCLIENT-781 . I can
take a look at this in the coming days -- do you think an implementation that just closes the prior connection & makes a new one on a subsequent request is OK, or does HttpClient need to close the connection immediately once the timeout elapses? (I don't think the latter is possible without an additional thread.) Sam On Wed, Jun 25, 2008 at 5:33 PM, Oleg Kalnichevski <olegk@...> wrote: > On Wed, 2008-06-25 at 17:14 -0400, Sam Berlin wrote: >> Hi Folks, >> >> We just stumbled across an issue with persistent connections where the >> server is sending a Connection: Keep-Alive header, but also a >> Keep-Alive header with a timeout=## (and a max=##). HttpClient's >> connection management sees that the connection can be kept alive, so >> pools it for potential reuse, but doesn't respect the timeout (or max) >> values. The problem is that the connection remains opens from the >> eyes of HttpClient. A subsequent request sends correctly (with no >> IOX's, oddly enough), but reading causes HttpClient to read a -1 and >> throw a NoHttpResponseException. This is against a run-of-the-mill >> Apache httpd server. >> >> This is normally supposed to be handled by a HttpRequestRetryHandler, >> to retry to the connection, but I'm wondering if it shouldn't even get >> there in the first place. Should HttpClient respect the Keep-Alive >> header's timeout (and max, or other) token and use a new connection in >> the first place? >> > > Hi Sam > > We certainly need to fix that. Could you please open a JIRA for this > issue? > > Oleg > >> Sam >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscribe@... >> For additional commands, e-mail: dev-help@... >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscribe@... > For additional commands, e-mail: dev-help@... > > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscribe@... For additional commands, e-mail: dev-help@... |
|
|
Re: Keep-Alive timeout & persistent connections
by olegk
::
Rate this Message:
On Wed, 2008-06-25 at 17:42 -0400, Sam Berlin wrote:
> Done @ https://issues.apache.org/jira/browse/HTTPCLIENT-781 . I can > take a look at this in the coming days -- do you think an > implementation that just closes the prior connection & makes a new one > on a subsequent request is OK, or does HttpClient need to close the > connection immediately once the timeout elapses? (I don't think the > latter is possible without an additional thread.) > I think the former is good enough, but a possibility to introduce a more sophisticated eviction policy would be an advantage. This however does not seem to require changes in the public API so it could be done after BETA1. What we need to do before the API freeze is to ensure there is a way to set a timeout value on individual managed connections. Oleg > Sam > > On Wed, Jun 25, 2008 at 5:33 PM, Oleg Kalnichevski <olegk@...> wrote: > > On Wed, 2008-06-25 at 17:14 -0400, Sam Berlin wrote: > >> Hi Folks, > >> > >> We just stumbled across an issue with persistent connections where the > >> server is sending a Connection: Keep-Alive header, but also a > >> Keep-Alive header with a timeout=## (and a max=##). HttpClient's > >> connection management sees that the connection can be kept alive, so > >> pools it for potential reuse, but doesn't respect the timeout (or max) > >> values. The problem is that the connection remains opens from the > >> eyes of HttpClient. A subsequent request sends correctly (with no > >> IOX's, oddly enough), but reading causes HttpClient to read a -1 and > >> throw a NoHttpResponseException. This is against a run-of-the-mill > >> Apache httpd server. > >> > >> This is normally supposed to be handled by a HttpRequestRetryHandler, > >> to retry to the connection, but I'm wondering if it shouldn't even get > >> there in the first place. Should HttpClient respect the Keep-Alive > >> header's timeout (and max, or other) token and use a new connection in > >> the first place? > >> > > > > Hi Sam > > > > We certainly need to fix that. Could you please open a JIRA for this > > issue? > > > > Oleg > > > >> Sam > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: dev-unsubscribe@... > >> For additional commands, e-mail: dev-help@... > >> > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscribe@... > > For additional commands, e-mail: dev-help@... > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscribe@... > For additional commands, e-mail: dev-help@... > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscribe@... For additional commands, e-mail: dev-help@... |
|
|
Re: Keep-Alive timeout & persistent connections
by Sam Berlin
::
Rate this Message:
FYI, the fix is in there now (along with a bunch of new tests), and a
new 'closeExpiredConnections' method that can be used the same way closeIdleConnections works if you want to bulk-close all expired connections at any given moment. Sam On Wed, Jun 25, 2008 at 5:52 PM, Oleg Kalnichevski <olegk@...> wrote: > On Wed, 2008-06-25 at 17:42 -0400, Sam Berlin wrote: >> Done @ https://issues.apache.org/jira/browse/HTTPCLIENT-781 . I can >> take a look at this in the coming days -- do you think an >> implementation that just closes the prior connection & makes a new one >> on a subsequent request is OK, or does HttpClient need to close the >> connection immediately once the timeout elapses? (I don't think the >> latter is possible without an additional thread.) >> > > I think the former is good enough, but a possibility to introduce a more > sophisticated eviction policy would be an advantage. This however does > not seem to require changes in the public API so it could be done after > BETA1. What we need to do before the API freeze is to ensure there is a > way to set a timeout value on individual managed connections. > > Oleg > > >> Sam >> >> On Wed, Jun 25, 2008 at 5:33 PM, Oleg Kalnichevski <olegk@...> wrote: >> > On Wed, 2008-06-25 at 17:14 -0400, Sam Berlin wrote: >> >> Hi Folks, >> >> >> >> We just stumbled across an issue with persistent connections where the >> >> server is sending a Connection: Keep-Alive header, but also a >> >> Keep-Alive header with a timeout=## (and a max=##). HttpClient's >> >> connection management sees that the connection can be kept alive, so >> >> pools it for potential reuse, but doesn't respect the timeout (or max) >> >> values. The problem is that the connection remains opens from the >> >> eyes of HttpClient. A subsequent request sends correctly (with no >> >> IOX's, oddly enough), but reading causes HttpClient to read a -1 and >> >> throw a NoHttpResponseException. This is against a run-of-the-mill >> >> Apache httpd server. >> >> >> >> This is normally supposed to be handled by a HttpRequestRetryHandler, >> >> to retry to the connection, but I'm wondering if it shouldn't even get >> >> there in the first place. Should HttpClient respect the Keep-Alive >> >> header's timeout (and max, or other) token and use a new connection in >> >> the first place? >> >> >> > >> > Hi Sam >> > >> > We certainly need to fix that. Could you please open a JIRA for this >> > issue? >> > >> > Oleg >> > >> >> Sam >> >> >> >> --------------------------------------------------------------------- >> >> To unsubscribe, e-mail: dev-unsubscribe@... >> >> For additional commands, e-mail: dev-help@... >> >> >> > >> > >> > --------------------------------------------------------------------- >> > To unsubscribe, e-mail: dev-unsubscribe@... >> > For additional commands, e-mail: dev-help@... >> > >> > >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscribe@... >> For additional commands, e-mail: dev-help@... >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscribe@... > For additional commands, e-mail: dev-help@... > > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscribe@... For additional commands, e-mail: dev-help@... |
|
|
|
|
|
Re: Keep-Alive timeout & persistent connections
by Sam Berlin
::
Rate this Message:
Yikes, that's no good. Did you test this using the first revision of
the fix or the second (changed to use 'setIdleDuration' in ManagedClientConnection)? Think it'd be possible to isolate the problem to a unit test? Sam On Mon, Jun 30, 2008 at 10:33 AM, Paul Lindner <plindner@...> wrote: > > I tested this code using production traffic and the results are not so good. > Ended up with a connection/memory leak. The jmap histogram of a server in > this bad state is attached. > > Also, I had some issues with one host, it used a keepalive with a timeout=1. > It appears that the route pool filled up with connections. It also appears > that when it closes an expired free connection that it is not released back > into the pool... Here's the debug output showing the final connection > allocation and the attempt to get a connection from a full pool. It appears > that Freeing the connection puts it back on the Idle pool, but it becomes > inaccessible for some reason. > > > 2008-06-29 19:22:27,761 [http-80-55] DEBUG > org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager - > ThreadSafeClientConnManager.getConnection: > HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com], timeout = > 50002008-06-29 19:22:27,761 [http-80-55] DEBUG > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Getting free connection > [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null] > 2008-06-29 19:22:27,761 [http-80-55] DEBUG > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Closing expired free > connection [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null] > 2008-06-29 19:22:27,761 [http-80-55] DEBUG > org.apache.http.impl.conn.DefaultClientConnection - Connection closed > 2008-06-29 19:22:27,761 [http-80-55] DEBUG > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - No free connections > [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null] > 2008-06-29 19:22:27,761 [http-80-55] DEBUG > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Available capacity: 1 > [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null] > 2008-06-29 19:22:27,761 [http-80-55] DEBUG > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Creating new connection > [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]] > 2008-06-29 19:22:27,761 [http-80-55] DEBUG > org.apache.http.impl.conn.DefaultClientConnection - Connection closed > 2008-06-29 19:22:27,818 [http-80-55] DEBUG > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Freeing connection > [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null] > 2008-06-29 19:22:27,818 [http-80-55] DEBUG > org.apache.http.impl.conn.IdleConnectionHandler - Adding connection at: > 12147925478182008-06-29 19:22:27,818 > [http-80-55] DEBUG org.apache.http.impl.conn.tsccm.ConnPoolByRoute - > Notifying no-one, there are no waiting threads > > ----------------------------------------------- > 2008-06-29 19:22:29,274 [http-80-26] DEBUG > org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager - > ThreadSafeClientConnManager.getConnection: > HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com], timeout = 5000 > 2008-06-29 19:22:29,274 [http-80-26] DEBUG > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Getting free connection > [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null] > 2008-06-29 19:22:29,274 [http-80-26] DEBUG > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Closing expired free > connection [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null] > 2008-06-29 19:22:29,274 [http-80-26] DEBUG > org.apache.http.impl.conn.DefaultClientConnection - Connection closed > 2008-06-29 19:22:29,274 [http-80-26] DEBUG > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - No free connections > [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null] > 2008-06-29 19:22:29,274 [http-80-26] DEBUG > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Available capacity: 0 > [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null] > 2008-06-29 19:22:29,274 [http-80-26] DEBUG > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Need to wait for > connection [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null] > 2008-06-29 19:22:29,331 [http-80-79] DEBUG > org.apache.http.impl.conn.DefaultClientConnection - Connection shut down > 2008-06-29 19:22:29,331 [http-80-79] DEBUG > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Freeing connection > [HttpRoute[{}->http://ads.socialmedia.com]][null] > 2008-06-29 19:22:29,331 [http-80-79] DEBUG > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Notifying thread waiting > on any pool > 2008-06-29 19:22:29,331 [http-80-26] DEBUG > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - No free connections > [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null] > 2008-06-29 19:22:29,331 [http-80-26] DEBUG > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Available capacity: 0 > [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null] > 2008-06-29 19:22:29,331 [http-80-26] DEBUG > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Need to wait for > connection [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null] > 2008-06-29 19:22:29,331 [http-80-26] DEBUG > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - No free connections > [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null] > 2008-06-29 19:22:29,331 [http-80-26] DEBUG > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Available capacity: 0 > [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null] > 2008-06-29 19:22:29,331 [http-80-26] DEBUG > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Need to wait for > connection [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null] > > > > Size Count Class description > ------------------------------------------------------- > 684164112 558913 byte[] > 492584016 2134710 char[] > 209551728 4365661 java.util.concurrent.locks.ReentrantLock$NonfairSync > 209517696 4364952 java.util.concurrent.ConcurrentHashMap$Segment > 144470200 4364956 java.util.concurrent.ConcurrentHashMap$HashEntry[] > 101379880 2534497 java.lang.String > 65693112 628849 java.lang.Object[] > 47067200 588340 com.friend.data.UserAppBean > 41467120 272810 java.util.concurrent.ConcurrentHashMap$Segment[] > 37081896 272661 org.apache.catalina.session.StandardSession > 32830608 683971 java.util.HashMap$Entry > 30965200 275343 java.util.Hashtable$Entry[] > 24135760 603394 java.util.ArrayList > 22612816 115641 java.util.HashMap$Entry[] > 19871936 620998 java.util.Date > 19642320 272810 java.util.concurrent.ConcurrentHashMap > 18644208 224156 int[] > 17608256 275129 java.util.Hashtable > 14569296 303527 java.util.concurrent.ConcurrentHashMap$HashEntry > 13089600 272700 java.beans.PropertyChangeSupport > 13054072 97229 * ConstMethodKlass > 12340480 154256 org.apache.tomcat.util.buf.MessageBytes > 11679960 97229 * MethodKlass > 10593856 165529 org.apache.tomcat.util.buf.ByteChunk > 10529104 10652 * ConstantPoolKlass > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscribe@... > For additional commands, e-mail: dev-help@... > > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscribe@... For additional commands, e-mail: dev-help@... |
|
|
Re: Keep-Alive timeout & persistent connections
by Paul Lindner-2
::
Rate this Message:
The tests below were using the latest trunk code (r672641)
I reverted back to r670018, where things are stable. I can try and see if I can reproduce it, but it may take some time. If anyone wants to try this it should be a simple matter. Configure an apache server to return headers like this: Keep-Alive: timeout=1, max=100 Connection: Keep-Alive And then use the threaded client to make simultaneous connections to it. On Jun 30, 2008, at 7:43 AM, Sam Berlin wrote: > Yikes, that's no good. Did you test this using the first revision of > the fix or the second (changed to use 'setIdleDuration' in > ManagedClientConnection)? > > Think it'd be possible to isolate the problem to a unit test? > > Sam > > On Mon, Jun 30, 2008 at 10:33 AM, Paul Lindner <plindner@...> > wrote: >> >> I tested this code using production traffic and the results are not >> so good. >> Ended up with a connection/memory leak. The jmap histogram of a >> server in >> this bad state is attached. >> >> Also, I had some issues with one host, it used a keepalive with a >> timeout=1. >> It appears that the route pool filled up with connections. It also >> appears >> that when it closes an expired free connection that it is not >> released back >> into the pool... Here's the debug output showing the final >> connection >> allocation and the attempt to get a connection from a full pool. >> It appears >> that Freeing the connection puts it back on the Idle pool, but it >> becomes >> inaccessible for some reason. >> >> >> 2008-06-29 19:22:27,761 [http-80-55] DEBUG >> org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager - >> ThreadSafeClientConnManager.getConnection: >> HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com], timeout = >> 50002008-06-29 19:22:27,761 [http-80-55] DEBUG >> org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Getting free >> connection >> [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null] >> 2008-06-29 19:22:27,761 [http-80-55] DEBUG >> org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Closing expired >> free >> connection [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]] >> [null] >> 2008-06-29 19:22:27,761 [http-80-55] DEBUG >> org.apache.http.impl.conn.DefaultClientConnection - Connection >> closed >> 2008-06-29 19:22:27,761 [http-80-55] DEBUG >> org.apache.http.impl.conn.tsccm.ConnPoolByRoute - No free >> connections >> [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null] >> 2008-06-29 19:22:27,761 [http-80-55] DEBUG >> org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Available >> capacity: 1 >> [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null] >> 2008-06-29 19:22:27,761 [http-80-55] DEBUG >> org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Creating new >> connection >> [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]] >> 2008-06-29 19:22:27,761 [http-80-55] DEBUG >> org.apache.http.impl.conn.DefaultClientConnection - Connection >> closed >> 2008-06-29 19:22:27,818 [http-80-55] DEBUG >> org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Freeing connection >> [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null] >> 2008-06-29 19:22:27,818 [http-80-55] DEBUG >> org.apache.http.impl.conn.IdleConnectionHandler - Adding >> connection at: >> 12147925478182008-06-29 19:22:27,818 >> [http-80-55] DEBUG org.apache.http.impl.conn.tsccm.ConnPoolByRoute - >> Notifying no-one, there are no waiting threads >> >> ----------------------------------------------- >> 2008-06-29 19:22:29,274 [http-80-26] DEBUG >> org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager - >> ThreadSafeClientConnManager.getConnection: >> HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com], timeout = 5000 >> 2008-06-29 19:22:29,274 [http-80-26] DEBUG >> org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Getting free >> connection >> [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null] >> 2008-06-29 19:22:29,274 [http-80-26] DEBUG >> org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Closing expired >> free >> connection [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]] >> [null] >> 2008-06-29 19:22:29,274 [http-80-26] DEBUG >> org.apache.http.impl.conn.DefaultClientConnection - Connection >> closed >> 2008-06-29 19:22:29,274 [http-80-26] DEBUG >> org.apache.http.impl.conn.tsccm.ConnPoolByRoute - No free >> connections >> [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null] >> 2008-06-29 19:22:29,274 [http-80-26] DEBUG >> org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Available >> capacity: 0 >> [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null] >> 2008-06-29 19:22:29,274 [http-80-26] DEBUG >> org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Need to wait for >> connection [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]] >> [null] >> 2008-06-29 19:22:29,331 [http-80-79] DEBUG >> org.apache.http.impl.conn.DefaultClientConnection - Connection >> shut down >> 2008-06-29 19:22:29,331 [http-80-79] DEBUG >> org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Freeing connection >> [HttpRoute[{}->http://ads.socialmedia.com]][null] >> 2008-06-29 19:22:29,331 [http-80-79] DEBUG >> org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Notifying thread >> waiting >> on any pool >> 2008-06-29 19:22:29,331 [http-80-26] DEBUG >> org.apache.http.impl.conn.tsccm.ConnPoolByRoute - No free >> connections >> [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null] >> 2008-06-29 19:22:29,331 [http-80-26] DEBUG >> org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Available >> capacity: 0 >> [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null] >> 2008-06-29 19:22:29,331 [http-80-26] DEBUG >> org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Need to wait for >> connection [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]] >> [null] >> 2008-06-29 19:22:29,331 [http-80-26] DEBUG >> org.apache.http.impl.conn.tsccm.ConnPoolByRoute - No free >> connections >> [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null] >> 2008-06-29 19:22:29,331 [http-80-26] DEBUG >> org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Available >> capacity: 0 >> [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null] >> 2008-06-29 19:22:29,331 [http-80-26] DEBUG >> org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Need to wait for >> connection [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]] >> [null] >> >> >> >> Size Count Class description >> ------------------------------------------------------- >> 684164112 558913 byte[] >> 492584016 2134710 char[] >> 209551728 4365661 java.util.concurrent.locks.ReentrantLock >> $NonfairSync >> 209517696 4364952 java.util.concurrent.ConcurrentHashMap >> $Segment >> 144470200 4364956 java.util.concurrent.ConcurrentHashMap >> $HashEntry[] >> 101379880 2534497 java.lang.String >> 65693112 628849 java.lang.Object[] >> 47067200 588340 com.friend.data.UserAppBean >> 41467120 272810 java.util.concurrent.ConcurrentHashMap >> $Segment[] >> 37081896 272661 org.apache.catalina.session.StandardSession >> 32830608 683971 java.util.HashMap$Entry >> 30965200 275343 java.util.Hashtable$Entry[] >> 24135760 603394 java.util.ArrayList >> 22612816 115641 java.util.HashMap$Entry[] >> 19871936 620998 java.util.Date >> 19642320 272810 java.util.concurrent.ConcurrentHashMap >> 18644208 224156 int[] >> 17608256 275129 java.util.Hashtable >> 14569296 303527 java.util.concurrent.ConcurrentHashMap >> $HashEntry >> 13089600 272700 java.beans.PropertyChangeSupport >> 13054072 97229 * ConstMethodKlass >> 12340480 154256 org.apache.tomcat.util.buf.MessageBytes >> 11679960 97229 * MethodKlass >> 10593856 165529 org.apache.tomcat.util.buf.ByteChunk >> 10529104 10652 * ConstantPoolKlass >> >> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscribe@... >> For additional commands, e-mail: dev-help@... >> >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscribe@... > For additional commands, e-mail: dev-help@... > Paul Lindner plindner@... --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscribe@... For additional commands, e-mail: dev-help@... |
|
|
Re: Keep-Alive timeout & persistent connections
by olegk
::
Rate this Message:
On Mon, 2008-06-30 at 10:43 -0400, Sam Berlin wrote:
> Yikes, that's no good. Did you test this using the first revision of > the fix or the second (changed to use 'setIdleDuration' in > ManagedClientConnection)? > > Think it'd be possible to isolate the problem to a unit test? > > Sam > Sam, Maybe unrelated to the resource leak but something that struck me as odd. Currently AbstractClientConnAdapter starts counting the idle time from the moment of receiving the last header. This does not seem right to me. I guess we should start counting from the moment of the last I/O on that connection. For simplicity reasons I think we can assume this to be the moment the connection is released back to the manager, as connections get released automatically immediately when the data stream returns EOF. Does this make sense? Oleg > On Mon, Jun 30, 2008 at 10:33 AM, Paul Lindner <plindner@...> wrote: > > > > I tested this code using production traffic and the results are not so good. > > Ended up with a connection/memory leak. The jmap histogram of a server in > > this bad state is attached. > > > > Also, I had some issues with one host, it used a keepalive with a timeout=1. > > It appears that the route pool filled up with connections. It also appears > > that when it closes an expired free connection that it is not released back > > into the pool... Here's the debug output showing the final connection > > allocation and the attempt to get a connection from a full pool. It appears > > that Freeing the connection puts it back on the Idle pool, but it becomes > > inaccessible for some reason. > > > > > > 2008-06-29 19:22:27,761 [http-80-55] DEBUG > > org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager - > > ThreadSafeClientConnManager.getConnection: > > HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com], timeout = > > 50002008-06-29 19:22:27,761 [http-80-55] DEBUG > > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Getting free connection > > [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null] > > 2008-06-29 19:22:27,761 [http-80-55] DEBUG > > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Closing expired free > > connection [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null] > > 2008-06-29 19:22:27,761 [http-80-55] DEBUG > > org.apache.http.impl.conn.DefaultClientConnection - Connection closed > > 2008-06-29 19:22:27,761 [http-80-55] DEBUG > > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - No free connections > > [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null] > > 2008-06-29 19:22:27,761 [http-80-55] DEBUG > > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Available capacity: 1 > > [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null] > > 2008-06-29 19:22:27,761 [http-80-55] DEBUG > > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Creating new connection > > [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]] > > 2008-06-29 19:22:27,761 [http-80-55] DEBUG > > org.apache.http.impl.conn.DefaultClientConnection - Connection closed > > 2008-06-29 19:22:27,818 [http-80-55] DEBUG > > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Freeing connection > > [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null] > > 2008-06-29 19:22:27,818 [http-80-55] DEBUG > > org.apache.http.impl.conn.IdleConnectionHandler - Adding connection at: > > 12147925478182008-06-29 19:22:27,818 > > [http-80-55] DEBUG org.apache.http.impl.conn.tsccm.ConnPoolByRoute - > > Notifying no-one, there are no waiting threads > > > > ----------------------------------------------- > > 2008-06-29 19:22:29,274 [http-80-26] DEBUG > > org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager - > > ThreadSafeClientConnManager.getConnection: > > HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com], timeout = 5000 > > 2008-06-29 19:22:29,274 [http-80-26] DEBUG > > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Getting free connection > > [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null] > > 2008-06-29 19:22:29,274 [http-80-26] DEBUG > > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Closing expired free > > connection [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null] > > 2008-06-29 19:22:29,274 [http-80-26] DEBUG > > org.apache.http.impl.conn.DefaultClientConnection - Connection closed > > 2008-06-29 19:22:29,274 [http-80-26] DEBUG > > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - No free connections > > [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null] > > 2008-06-29 19:22:29,274 [http-80-26] DEBUG > > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Available capacity: 0 > > [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null] > > 2008-06-29 19:22:29,274 [http-80-26] DEBUG > > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Need to wait for > > connection [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null] > > 2008-06-29 19:22:29,331 [http-80-79] DEBUG > > org.apache.http.impl.conn.DefaultClientConnection - Connection shut down > > 2008-06-29 19:22:29,331 [http-80-79] DEBUG > > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Freeing connection > > [HttpRoute[{}->http://ads.socialmedia.com]][null] > > 2008-06-29 19:22:29,331 [http-80-79] DEBUG > > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Notifying thread waiting > > on any pool > > 2008-06-29 19:22:29,331 [http-80-26] DEBUG > > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - No free connections > > [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null] > > 2008-06-29 19:22:29,331 [http-80-26] DEBUG > > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Available capacity: 0 > > [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null] > > 2008-06-29 19:22:29,331 [http-80-26] DEBUG > > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Need to wait for > > connection [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null] > > 2008-06-29 19:22:29,331 [http-80-26] DEBUG > > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - No free connections > > [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null] > > 2008-06-29 19:22:29,331 [http-80-26] DEBUG > > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Available capacity: 0 > > [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null] > > 2008-06-29 19:22:29,331 [http-80-26] DEBUG > > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Need to wait for > > connection [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null] > > > > > > > > Size Count Class description > > ------------------------------------------------------- > > 684164112 558913 byte[] > > 492584016 2134710 char[] > > 209551728 4365661 java.util.concurrent.locks.ReentrantLock$NonfairSync > > 209517696 4364952 java.util.concurrent.ConcurrentHashMap$Segment > > 144470200 4364956 java.util.concurrent.ConcurrentHashMap$HashEntry[] > > 101379880 2534497 java.lang.String > > 65693112 628849 java.lang.Object[] > > 47067200 588340 com.friend.data.UserAppBean > > 41467120 272810 java.util.concurrent.ConcurrentHashMap$Segment[] > > 37081896 272661 org.apache.catalina.session.StandardSession > > 32830608 683971 java.util.HashMap$Entry > > 30965200 275343 java.util.Hashtable$Entry[] > > 24135760 603394 java.util.ArrayList > > 22612816 115641 java.util.HashMap$Entry[] > > 19871936 620998 java.util.Date > > 19642320 272810 java.util.concurrent.ConcurrentHashMap > > 18644208 224156 int[] > > 17608256 275129 java.util.Hashtable > > 14569296 303527 java.util.concurrent.ConcurrentHashMap$HashEntry > > 13089600 272700 java.beans.PropertyChangeSupport > > 13054072 97229 * ConstMethodKlass > > 12340480 154256 org.apache.tomcat.util.buf.MessageBytes > > 11679960 97229 * MethodKlass > > 10593856 165529 org.apache.tomcat.util.buf.ByteChunk > > 10529104 10652 * ConstantPoolKlass > > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscribe@... > > For additional commands, e-mail: dev-help@... > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscribe@... > For additional commands, e-mail: dev-help@... > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscribe@... For additional commands, e-mail: dev-help@... |
|
|
Re: Keep-Alive timeout & persistent connections
by Sam Berlin
::
Rate this Message:
Yeah, that makes sense. Dunno what was going through my head at the
time, but for some reason I thought it made sense to calculate the time when the headers are read. It definitely simplifies the code to do it just when the connection is released. Sam On Mon, Jun 30, 2008 at 2:51 PM, Oleg Kalnichevski <olegk@...> wrote: > On Mon, 2008-06-30 at 10:43 -0400, Sam Berlin wrote: >> Yikes, that's no good. Did you test this using the first revision of >> the fix or the second (changed to use 'setIdleDuration' in >> ManagedClientConnection)? >> >> Think it'd be possible to isolate the problem to a unit test? >> >> Sam >> > > Sam, > > Maybe unrelated to the resource leak but something that struck me as > odd. Currently AbstractClientConnAdapter starts counting the idle time > from the moment of receiving the last header. This does not seem right > to me. I guess we should start counting from the moment of the last I/O > on that connection. For simplicity reasons I think we can assume this to > be the moment the connection is released back to the manager, as > connections get released automatically immediately when the data stream > returns EOF. > > Does this make sense? > > Oleg > > >> On Mon, Jun 30, 2008 at 10:33 AM, Paul Lindner <plindner@...> wrote: >> > >> > I tested this code using production traffic and the results are not so good. >> > Ended up with a connection/memory leak. The jmap histogram of a server in >> > this bad state is attached. >> > >> > Also, I had some issues with one host, it used a keepalive with a timeout=1. >> > It appears that the route pool filled up with connections. It also appears >> > that when it closes an expired free connection that it is not released back >> > into the pool... Here's the debug output showing the final connection >> > allocation and the attempt to get a connection from a full pool. It appears >> > that Freeing the connection puts it back on the Idle pool, but it becomes >> > inaccessible for some reason. >> > >> > >> > 2008-06-29 19:22:27,761 [http-80-55] DEBUG >> > org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager - >> > ThreadSafeClientConnManager.getConnection: >> > HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com], timeout = >> > 50002008-06-29 19:22:27,761 [http-80-55] DEBUG >> > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Getting free connection >> > [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null] >> > 2008-06-29 19:22:27,761 [http-80-55] DEBUG >> > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Closing expired free >> > connection [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null] >> > 2008-06-29 19:22:27,761 [http-80-55] DEBUG >> > org.apache.http.impl.conn.DefaultClientConnection - Connection closed >> > 2008-06-29 19:22:27,761 [http-80-55] DEBUG >> > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - No free connections >> > [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null] >> > 2008-06-29 19:22:27,761 [http-80-55] DEBUG >> > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Available capacity: 1 >> > [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null] >> > 2008-06-29 19:22:27,761 [http-80-55] DEBUG >> > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Creating new connection >> > [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]] > |
