NMS and network connectivity

[mkeenan]

Here's my setup:
ActiveMQ 5.1
NMS 1.0 with connection URL: tcp://machinename:61616?jms.prefetchPolicy.all=1


I'm having a problem with AMQ/NMS when the network connection is lost. To simulate this problem I manually disable/enable the network interface on my local box during the course of the test.

When the network connection is lost, the client enters a loop until the connection is re-established. However, even after the client creates a new connection, session, consumer, and registers a listener no messages are received.  In order to receive message I had to restart the broker.

I used the JMX console and notice that the old connection is still listed, which is reasonable because the client is not able to shut down cleanly when the network unexpectedly goes down.

I also observed that if I manually close the old connection (using the JMX console) the client does begin to receive messages.

What can I do to fix this situation?  It's not practical to manually restart the broker or manually manage connections when we have network blips.

[semog]

Hi,
We had a similar problem as well.  We called them "zombie connections".  I
think we solved this by moving to ActiveMQ 5.2. If that doesn't work, you
can turn on the timeout inactivity option by setting the
"wireFormat.MaxInactivityDuration=nnnn"
connection option.  This option is in the NMS 1.1 trunk, though.  I don't
think I ported it to the 1.0.0 branch.

- Jim

On Wed, Oct 8, 2008 at 9:14 AM, mkeenan <keenan@...> wrote:

>
> Here's my setup:
> ActiveMQ 5.1
> NMS 1.0 with connection URL:
> tcp://machinename:61616?jms.prefetchPolicy.all=1
>
>
> I'm having a problem with AMQ/NMS when the network connection is lost. To
> simulate this problem I manually disable/enable the network interface on my
> local box during the course of the test.
>
> When the network connection is lost, the client enters a loop until the
> connection is re-established. However, even after the client creates a new
> connection, session, consumer, and registers a listener no messages are
> received.  In order to receive message I had to restart the broker.
>
> I used the JMX console and notice that the old connection is still listed,
> which is reasonable because the client is not able to shut down cleanly
> when
> the network unexpectedly goes down.
>
> I also observed that if I manually close the old connection (using the JMX
> console) the client does begin to receive messages.
>
> What can I do to fix this situation?  It's not practical to manually
> restart
> the broker or manually manage connections when we have network blips.
>
>
> --
> View this message in context:
> http://www.nabble.com/NMS-and-network-connectivity-tp19882147p19882147.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>

[mkeenan]

Jim,

Thanks for responding. I did try the same test under AMQ 5.2 and 5.3, but the result was the same each time. I will try the 1.1 release with the parameter you suggest and report back.  

One follow up question: I thought the MaxInactivityDuration was already defaulted to 30 seconds:

http://activemq.apache.org/configuring-wire-formats.html

Is there a need to set it explicitly?

semog wrote:

Hi,
We had a similar problem as well.  We called them "zombie connections".  I
think we solved this by moving to ActiveMQ 5.2. If that doesn't work, you
can turn on the timeout inactivity option by setting the
"wireFormat.MaxInactivityDuration=nnnn"
connection option.  This option is in the NMS 1.1 trunk, though.  I don't
think I ported it to the 1.0.0 branch.

- Jim

On Wed, Oct 8, 2008 at 9:14 AM, mkeenan <[e-mail]> wrote:

>
> Here's my setup:
> ActiveMQ 5.1
> NMS 1.0 with connection URL:
> tcp://machinename:61616?jms.prefetchPolicy.all=1
>
>
> I'm having a problem with AMQ/NMS when the network connection is lost. To
> simulate this problem I manually disable/enable the network interface on my
> local box during the course of the test.
>
> When the network connection is lost, the client enters a loop until the
> connection is re-established. However, even after the client creates a new
> connection, session, consumer, and registers a listener no messages are
> received.  In order to receive message I had to restart the broker.
>
> I used the JMX console and notice that the old connection is still listed,
> which is reasonable because the client is not able to shut down cleanly
> when
> the network unexpectedly goes down.
>
> I also observed that if I manually close the old connection (using the JMX
> console) the client does begin to receive messages.
>
> What can I do to fix this situation?  It's not practical to manually
> restart
> the broker or manually manage connections when we have network blips.
>
>
> --
> View this message in context:
> http://www.nabble.com/NMS-and-network-connectivity-tp19882147p19882147.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>

[mkeenan]

Apologies for responding to myself, but I wanted to mention that I did try the 1.1 NMS client with the  "wireFormat.MaxInactivityDuration=nnnn" connection option, and it did resolve the issues we had with zombie/orphan connections on the broker.

[semog]

Thanks for testing this.  I also saw your follow up reply where the
maxInactivityDuration option solved the zombie issue.  Looking at the wire
format page you linked, you are correct.  The option should be turned on by
default.  I will log a JIRA to have it turned on to 30 seconds by default.
Best,
Jim

On Thu, Oct 9, 2008 at 8:57 AM, mkeenan <keenan@...> wrote:

>
> Jim,
>
> Thanks for responding. I did try the same test under AMQ 5.2 and 5.3, but
> the result was the same each time. I will try the 1.1 release with the
> parameter you suggest and report back.
>
> One follow up question: I thought the MaxInactivityDuration was already
> defaulted to 30 seconds:
>
> http://activemq.apache.org/configuring-wire-formats.html
>
> Is there a need to set it explicitly?
>
>
>
> semog wrote:
> >
> > Hi,
> > We had a similar problem as well.  We called them "zombie connections".
>  I
> > think we solved this by moving to ActiveMQ 5.2. If that doesn't work, you
> > can turn on the timeout inactivity option by setting the
> > "wireFormat.MaxInactivityDuration=nnnn"
> > connection option.  This option is in the NMS 1.1 trunk, though.  I don't
> > think I ported it to the 1.0.0 branch.
> >
> > - Jim
> >
> > On Wed, Oct 8, 2008 at 9:14 AM, mkeenan <keenan@...> wrote:
> >
> >>
> >> Here's my setup:
> >> ActiveMQ 5.1
> >> NMS 1.0 with connection URL:
> >> tcp://machinename:61616?jms.prefetchPolicy.all=1
> >>
> >>
> >> I'm having a problem with AMQ/NMS when the network connection is lost.
> To
> >> simulate this problem I manually disable/enable the network interface on
> >> my
> >> local box during the course of the test.
> >>
> >> When the network connection is lost, the client enters a loop until the
> >> connection is re-established. However, even after the client creates a
> >> new
> >> connection, session, consumer, and registers a listener no messages are
> >> received.  In order to receive message I had to restart the broker.
> >>
> >> I used the JMX console and notice that the old connection is still
> >> listed,
> >> which is reasonable because the client is not able to shut down cleanly
> >> when
> >> the network unexpectedly goes down.
> >>
> >> I also observed that if I manually close the old connection (using the
> >> JMX
> >> console) the client does begin to receive messages.
> >>
> >> What can I do to fix this situation?  It's not practical to manually
> >> restart
> >> the broker or manually manage connections when we have network blips.
> >>
> >>
> >> --
> >> View this message in context:
> >>
> http://www.nabble.com/NMS-and-network-connectivity-tp19882147p19882147.html
> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >>
> >>
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/NMS-and-network-connectivity-tp19882147p19901917.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>