EVMS problem

View: New views
3 Messages — Rating Filter:   Alert me  

EVMS problem

by hardhead :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message


I have a RAID5 managed by EVMS using LVM2 running on a Linux server.

Earlier today the system went down unexpectedly and when the box came up one of the disks had a bad block so the raid is running in degraded mode.  Unfortunately, when I try to add a spare disk to the array--I get a failure message:

Oct 06 23:16:02 jumbo _0_ Engine: engine_user_message: Message is: Engine: Plug-in MDRaid5RegMgr returned error 19 when committing changes for region md/md1 during phase 3.

I began to research this error message--and I *think* the problem is the MD object isn't activating.  Without that activation, I can not manage the RAID (so I can't add a spare).  When running a simple activate command in the CLI, the following appears in my evms-engine.log file:

_5_ Engine: is_object_change_pending: Change pending: Object lvm2/media/home needs to be activated.
_5_ Engine: is_object_change_pending: Change pending: Object lvm2/media/media needs to be activated.
_5_ Engine: is_object_change_pending: Change pending: Object md/md1 needs to be activated.
_5_ Engine: is_volume_change_pending: Change pending: Volume /dev/evms/home needs to be activated.
_5_ Engine: is_volume_change_pending: Change pending: Volume /dev/evms/media needs to be activated.
_3_ Engine: engine_ioctl_object: ioctl to object md/md1 failed with error code 5: Input/output error
_2_ MDRaid5RegMgr: md_ioctl_run_array: md_ioctl_run_array: IOCTL failed, region(md/md1) rc(5)
_3_ Engine: activate_object: Error code 5 activating object md/md1: Input/output error
_3_ Engine: engine_ioctl_object: ioctl to object md/md1 failed with error code 16: Device or resource busy

In /var/log/messages I see lots of:

md: array md1 already has disks!

When I run mdadm --detail /dev/md1 I get:

/dev/md1:
        Version : 00.90.03
  Creation Time : Mon May 15 20:55:42 2006
     Raid Level : raid5
    Device Size : 244195840 (232.88 GiB 250.06 GB)
   Raid Devices : 7
  Total Devices : 6
Preferred Minor : 1
    Persistence : Superblock is persistent

    Update Time : Sat Oct  6 22:26:37 2007
          State : active, degraded
 Active Devices : 6
Working Devices : 6
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-asymmetric
     Chunk Size : 256K

           UUID : 66774afe:e3d47cc5:57d24caa:8eb4f32b
         Events : 0.30322321

    Number   Major   Minor   RaidDevice State
       0     253        0        0      active sync   /dev/evms/.nodes/hdc1
       1       0        0        -      removed
       2     253        1        2      active sync   /dev/evms/.nodes/sdb1
       3     253        2        3      active sync   /dev/evms/.nodes/sdc1
       4     253        3        4      active sync   /dev/evms/.nodes/sdd1
       5     253        4        5      active sync   /dev/evms/.nodes/sde1
       6     253        5        6      active sync   /dev/evms/.nodes/sdf1

I have tried mdadm --stop /dev/md1 before running evms (or evmsn) and I keep running into the same problem.  Any help on this issue would be much appreciated.  I know I *could* potentially rebuild the raid with mdadm -a /dev/sda1, but I'm not sure how EVMS would react to that, and I'm not sure my two volumes stored on the raid (/dev/evms/media and /dev/evms/home) would remain intact. 

Also--when I run ls /dev/evms/.nodes I see:

hdc  hdc1  sda  sdb  sdb1  sdc  sdc1  sdd  sdd1  sde  sde1  sdf  sdf1

I notice sda1 doesn't yet exist, so I can't manually use mdadm to add sda1.

Please let me know if any more information would be helpful--and if there is any way to solve this problem.
Thanks,
-Michael Beam


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
Evms-devel mailing list
Evms-devel@...
To subscribe/unsubscribe, please visit:
https://lists.sourceforge.net/lists/listinfo/evms-devel

Re: EVMS problem

by Steinar H. Gunderson :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

On Sat, Oct 06, 2007 at 11:35:25PM -0400, hardhead wrote:
> I know I *could* potentially rebuild the raid with mdadm -a
> /dev/sda1, but I'm not sure how EVMS would react to that

FWIW, if you use /dev/evms/.nodes/sda1 instead, it's perfectly safe.

I recently saw your problem, but didn't find a good solution.

/* Steinar */
--
Homepage: http://www.sesse.net/

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
Evms-devel mailing list
Evms-devel@...
To subscribe/unsubscribe, please visit:
https://lists.sourceforge.net/lists/listinfo/evms-devel

Re: EVMS problem

by hardhead :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

I was able to fix the problem with a few steps with help from folks at the gentoo forums.  The discussion is detailed here:

http://forums.gentoo.org/viewtopic-t-595202.html

Ultimately, I found some discussion from the IRC logs that led to the commands that helped:

mdadm --stop /dev/md1
mdadm -A /dev/md1 -f /dev/evms/.nodes/hdc1 /dev/evms/.nodes/sdb1 /dev/evms/.nodes/sdc1 /dev/evms/.nodes/sdd1 /dev/evms/.nodes/sde1 /dev/evms/.nodes/sdf1

Cheers,
-Michael

On 10/8/07, Steinar H. Gunderson <sgunderson@... > wrote:
On Sat, Oct 06, 2007 at 11:35:25PM -0400, hardhead wrote:
> I know I *could* potentially rebuild the raid with mdadm -a
> /dev/sda1, but I'm not sure how EVMS would react to that

FWIW, if you use /dev/evms/.nodes/sda1 instead, it's perfectly safe.

I recently saw your problem, but didn't find a good solution.

/* Steinar */
--
Homepage: http://www.sesse.net/


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
Evms-devel mailing list
Evms-devel@...
To subscribe/unsubscribe, please visit:
https://lists.sourceforge.net/lists/listinfo/evms-devel
LightInTheBox - Buy quality products at wholesale price!