CARP not leaving backup state

[William Stuart]

(Sorry if this is a dupe, not sure if you had to be a subscriber to send
to the list)

Hello all,

I am a new to OpenBSD but not *nix in general...

I have two systems running OpenBSD 4.2.  It has 9 carp interfaces, and
has been running fine for months.  All of a sudden, both systems are in
BACKUP state.

I halted one of the systems then on the remaining system rebooted, shut
down and restarted, run "ifconfig carp1 state master", changed the
sysctls, removed the hostname files, rebooted, then replaced the
hostname files, fiddled with the advskew and lots of other things.

Even with no other system running, carp will not go into MASTER state,
period, no errors,  no logs.

I tried setting net.inet.carp.log=1 and 2 and 1000, I see no logs
anywhere in /var/log.

Anything else I can look at?

William

[Vinicius Vianna-2]

Hi William,

I don't know for sure, but I remember dealing with this kind of problem
and setting preempt did work, maybe worth a try:

/etc/sysctl.conf:
net.inet.carp.preempt=1

Anyone else?

HTH,
Vinicius

William Stuart escreveu:

> (Sorry if this is a dupe, not sure if you had to be a subscriber to
> send to the list)
>
> Hello all,
>
> I am a new to OpenBSD but not *nix in general...
>
> I have two systems running OpenBSD 4.2.  It has 9 carp interfaces, and
> has been running fine for months.  All of a sudden, both systems are
> in BACKUP state.
>
> I halted one of the systems then on the remaining system rebooted,
> shut down and restarted, run "ifconfig carp1 state master", changed
> the sysctls, removed the hostname files, rebooted, then replaced the
> hostname files, fiddled with the advskew and lots of other things.
>
> Even with no other system running, carp will not go into MASTER state,
> period, no errors,  no logs.
>
> I tried setting net.inet.carp.log=1 and 2 and 1000, I see no logs
> anywhere in /var/log.
>
> Anything else I can look at?
>
> William

[William Stuart]

Thank you for the idea.  I did try that, but it still did not become MASTER.

I had to give up on carp (and redundancy altogether) and run with
aliases for now.

Vinicius Vianna wrote:

> Hi William,
>
> I don't know for sure, but I remember dealing with this kind of
> problem and setting preempt did work, maybe worth a try:
>
> /etc/sysctl.conf:
> net.inet.carp.preempt=1
>
> Anyone else?
>
> HTH,
> Vinicius
>
> William Stuart escreveu:
>> (Sorry if this is a dupe, not sure if you had to be a subscriber to
>> send to the list)
>>
>> Hello all,
>>
>> I am a new to OpenBSD but not *nix in general...
>>
>> I have two systems running OpenBSD 4.2.  It has 9 carp interfaces,
>> and has been running fine for months.  All of a sudden, both systems
>> are in BACKUP state.
>>
>> I halted one of the systems then on the remaining system rebooted,
>> shut down and restarted, run "ifconfig carp1 state master", changed
>> the sysctls, removed the hostname files, rebooted, then replaced the
>> hostname files, fiddled with the advskew and lots of other things.
>>
>> Even with no other system running, carp will not go into MASTER
>> state, period, no errors,  no logs.
>>
>> I tried setting net.inet.carp.log=1 and 2 and 1000, I see no logs
>> anywhere in /var/log.
>>
>> Anything else I can look at?
>>
>> William

[Markus Wernig]

Hi

Are you sure that all the interfaces you have configured carp on have
link and can connect to each other? (I've seen similar behaviour caused
by defective NICs: receive buffer not receiving while send buffer still
sending - try ping on all interfaces) Is lo up? Is there any other
router on the same network segment that propagates the same VHID with a
better metric (i suppose that a VRRP router on the same net could cause
trouble if it uses the same VHIDs).

no answers, just questions, I know ...

hth /markus

[William Stuart]

Markus Wernig wrote:

> Hi
>
> Are you sure that all the interfaces you have configured carp on have
> link and can connect to each other? (I've seen similar behaviour
> caused by defective NICs: receive buffer not receiving while send
> buffer still sending - try ping on all interfaces) Is lo up? Is there
> any other router on the same network segment that propagates the same
> VHID with a better metric (i suppose that a VRRP router on the same
> net could cause trouble if it uses the same VHIDs).
>
> no answers, just questions, I know ...
>
> hth /markus

Don't mind the questions!  Just hoping someone will ask one where I say,
"I am such an IDIOT! Why didn't I think of that!"

Yes, it had link (I was ssh'ing to the configured interface).  They can
connect to each other (I would assume if they couldn't see each other I
would have a MASTER/MASTER issue and not a BACKUP/BACKUP issue), I ended
up halting one of them to see if I can get just one to go MASTER when it
was alone, no dice.  I did a tcpdump and did not see any other VRRP traffic.

[Markus Wernig]

If you tcpdump do you see any carp traffic at all (ip proto 112)? Upon
reboot? And you did enable carp preemption on both hosts (sysctl
net.inet.carp.preempt=1)?

[Jason Dixon]

On Fri, Jul 18, 2008 at 01:26:07PM -0700, William Stuart wrote:

> Markus Wernig wrote:
>> Hi
>>
>> Are you sure that all the interfaces you have configured carp on have  
>> link and can connect to each other? (I've seen similar behaviour  
>> caused by defective NICs: receive buffer not receiving while send  
>> buffer still sending - try ping on all interfaces) Is lo up? Is there  
>> any other router on the same network segment that propagates the same  
>> VHID with a better metric (i suppose that a VRRP router on the same  
>> net could cause trouble if it uses the same VHIDs).
>>
>> no answers, just questions, I know ...
>>
>> hth /markus
> Don't mind the questions!  Just hoping someone will ask one where I say,  
> "I am such an IDIOT! Why didn't I think of that!"
>
> Yes, it had link (I was ssh'ing to the configured interface).  They can  
> connect to each other (I would assume if they couldn't see each other I  
> would have a MASTER/MASTER issue and not a BACKUP/BACKUP issue), I ended  
> up halting one of them to see if I can get just one to go MASTER when it  
> was alone, no dice.  I did a tcpdump and did not see any other VRRP
> traffic.

Is there any chance another CARP segment has recently been added to the
same switch / stack?  I've seen CARP vhid's "leak" through to other
broadcast domains on some Avaya switches.

--
Jason Dixon
DixonGroup Consulting
http://www.dixongroup.net/

[William Stuart]

Thanks everyone I figured it out!

19:13:46.334037 CARPv2-advertise 36: vhid=50 advbase=1 advskew=0
demote=0 (DF) [tos 0x10]
19:13:46.334299 CARPv2-advertise 36: vhid=50 advbase=1 advskew=0
demote=0 (DF) [tos 0x10]

Something is mirroring and replaying all the packets back.

Grrr.  Must be a vmWare config issue.

William

Jason Dixon wrote:

> If you'd like to send me your full `ifconfig -a` and pf.conf from both
> systems I'll take a look.
>
> -J.
>
> On Fri, Jul 18, 2008 at 06:19:22PM -0700, William Stuart wrote:
>  
>> Not before the failure, but I did test negative demoting as a fix..
>>
>> -bash-3.2# ifconfig -g carp           carp: carp demote count 0
>>
>> Jason Dixon wrote:
>>    
>>> Any chance you've been adjusting your carpdemote counters?
>>>
>>> -J.
>>>
>>> On Fri, Jul 18, 2008 at 06:04:19PM -0700, William Stuart wrote:
>>>  
>>>      
>>>> Gotcha.  Nope, only one set of OpenBSD boxes.
>>>>
>>>> Just as a test, I replaced the carp config for one of the IP
>>>> addresses  and changed the VHID. still backup.
>>>>
>>>> -bash-3.2# ifconfig
>>>> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 33168
>>>>        groups: lo
>>>>        inet6 ::1 prefixlen 128
>>>>        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
>>>>        inet 127.0.0.1 netmask 0xff000000
>>>> em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
>>>>        lladdr 00:50:56:a0:4f:81
>>>>        groups: egress
>>>>        media: Ethernet autoselect (1000baseT full-duplex,master)
>>>>        status: active
>>>>        inet6 fe80::250:56ff:fea0:4f81%em0 prefixlen 64 scopeid 0x1
>>>>        inet 172.19.161.64 netmask 0xffffffff broadcast 172.19.161.64
>>>> enc0: flags=0<> mtu 1536
>>>> carp5: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
>>>>        lladdr 00:00:5e:00:01:32
>>>>        carp: BACKUP carpdev em0 vhid 50 advbase 1 advskew 0
>>>>        groups: carp
>>>>        inet6 fe80::200:5eff:fe00:132%carp5 prefixlen 64 scopeid 0x4
>>>>        inet 172.19.161.67 netmask 0xffffff00 broadcast 172.19.161.255
>>>> -bash-3.2# cat /etc/hostname.carp5
>>>> inet 172.19.161.67 255.255.255.0 172.19.161.255 vhid 50 carpdev em0
>>>> pass  blahblahblah
>>>>
>>>> Jason Dixon wrote:
>>>>    
>>>>        
>>>>> I didn't mean someone else had.  I meant that perhaps you setup another
>>>>> pair on another network, but on the same switch.  It happened to us, and
>>>>> even though we didn't see "leaking" carp/vrrp packets, the problem
>>>>> disappeared as soon as we changed the vhid to something unique.
>>>>>
>>>>> Oh well, try try again.
>>>>>
>>>>> -J.
>>>>>
>>>>> On Fri, Jul 18, 2008 at 03:58:56PM -0700, William Stuart wrote:
>>>>>        
>>>>>          
>>>>>> Nope.  Confirmed with tcpdump.  No other carp/vrrp.
>>>>>>
>>>>>> Plus, where I am, no one else can spell B-S-D let alone install
>>>>>> one, let  alone setup carp.
>>>>>>
>>>>>> William
>>>>>>
>>>>>> Jason Dixon wrote:
>>>>>>            
>>>>>>            
>>>>>>> On Fri, Jul 18, 2008 at 01:26:07PM -0700, William Stuart wrote:
>>>>>>>                  
>>>>>>>              
>>>>>>>> Markus Wernig wrote:
>>>>>>>>                        
>>>>>>>>                
>>>>>>>>> Hi
>>>>>>>>>
>>>>>>>>> Are you sure that all the interfaces you have configured
>>>>>>>>> carp on have  link and can connect to each other? (I've
>>>>>>>>> seen similar behaviour  caused by defective NICs: receive
>>>>>>>>> buffer not receiving while send  buffer still sending - try
>>>>>>>>> ping on all interfaces) Is lo up? Is there  any other
>>>>>>>>> router on the same network segment that propagates the same
>>>>>>>>>  VHID with a better metric (i suppose that a VRRP router on
>>>>>>>>> the same  net could cause trouble if it uses the same
>>>>>>>>> VHIDs).
>>>>>>>>>
>>>>>>>>> no answers, just questions, I know ...
>>>>>>>>>
>>>>>>>>> hth /markus
>>>>>>>>>                              
>>>>>>>>>                  
>>>>>>>> Don't mind the questions!  Just hoping someone will ask one
>>>>>>>> where I say,  "I am such an IDIOT! Why didn't I think of
>>>>>>>> that!"
>>>>>>>>
>>>>>>>> Yes, it had link (I was ssh'ing to the configured interface).
>>>>>>>>  They can  connect to each other (I would assume if they
>>>>>>>> couldn't see each other I  would have a MASTER/MASTER issue
>>>>>>>> and not a BACKUP/BACKUP issue), I ended  up halting one of
>>>>>>>> them to see if I can get just one to go MASTER when it  was
>>>>>>>> alone, no dice.  I did a tcpdump and did not see any other
>>>>>>>> VRRP traffic.
>>>>>>>>                        
>>>>>>>>                
>>>>>>> Is there any chance another CARP segment has recently been added to the
>>>>>>> same switch / stack?  I've seen CARP vhid's "leak" through to other
>>>>>>> broadcast domains on some Avaya switches.

[Stuart Henderson]

On 2008-07-19, William Stuart <william@...> wrote:

> Thanks everyone I figured it out!
>
> 19:13:46.334037 CARPv2-advertise 36: vhid=50 advbase=1 advskew=0
> demote=0 (DF) [tos 0x10]
> 19:13:46.334299 CARPv2-advertise 36: vhid=50 advbase=1 advskew=0
> demote=0 (DF) [tos 0x10]
>
> Something is mirroring and replaying all the packets back.
>
> Grrr.  Must be a vmWare config issue.

Anyone asking about any wierd problems, _please_ mention any VMs that
may be involved early on in the thread... as always, a dmesg would be
a good starting point.

[William Stuart]

Hello everyone,

I am sorry for not mentioning it was a vmWare instance.  The packet
replay seemed to be the culprit.

This occured when we moved the image to a vmWare host running vmWare ESX
3.5 from 3.0.  Our working theory is that under 3.5 pernicious mode
works differently than under 3.0 and replays all of the traffic.

Has anyone else experienced this problem?

Stuart Henderson wrote:

> On 2008-07-19, William Stuart <william@...> wrote:
>  
>> Thanks everyone I figured it out!
>>
>> 19:13:46.334037 CARPv2-advertise 36: vhid=50 advbase=1 advskew=0
>> demote=0 (DF) [tos 0x10]
>> 19:13:46.334299 CARPv2-advertise 36: vhid=50 advbase=1 advskew=0
>> demote=0 (DF) [tos 0x10]
>>
>> Something is mirroring and replaying all the packets back.
>>
>> Grrr.  Must be a vmWare config issue.
>>    
>
> Anyone asking about any wierd problems, _please_ mention any VMs that
> may be involved early on in the thread... as always, a dmesg would be
> a good starting point.

[Stuart Henderson]

On 2008-07-21, William Stuart <william@...> wrote:
> I am sorry for not mentioning it was a vmWare instance.  The packet
> replay seemed to be the culprit.
>
> This occured when we moved the image to a vmWare host running vmWare ESX
> 3.5 from 3.0.  Our working theory is that under 3.5 pernicious mode
> works differently than under 3.0 and replays all of the traffic.
>
> Has anyone else experienced this problem?

No vmware here, but I have noticed some emulator or other (maybe
it was simh) returning transmitted ethernet frames on the receive side.

Is it any better with vic(4)? (and can you check if basic functions
work on ESX 3.x so the manual page, which only mentions 2.x, can be
updated?)

[Janne Johansson]

William Stuart wrote:

> Hello everyone,
>
> I am sorry for not mentioning it was a vmWare instance.  The packet
> replay seemed to be the culprit.
>
> This occured when we moved the image to a vmWare host running vmWare ESX
> 3.5 from 3.0.  Our working theory is that under 3.5 pernicious mode
> works differently than under 3.0 and replays all of the traffic.
>
> Has anyone else experienced this problem?

We have also experienced problems with CARP when moving ESX from 3.0i to
3.5. No solution yet.

[Denis Fondras]

> We have also experienced problems with CARP when moving ESX from 3.0i to
> 3.5. No solution yet.
>

Have tried to tweak the vSwitch settings ? I remember I made CARP work
with 2 OpenBSD 4.2 VM on ESX 3.5 after changing some settings in the
networking properties... (I know I should document what I do...)

Denis