x86 Hardware RAID Traps with BS_RAID_CHK

The following script is designed to run on Solaris x86 or Redhat systems with LSI and Adaptec hardware controllers.  It will check for degraded states of those controllers using mpt-status, arccon or raidctl depending on hardware vendor and OS.   If a degraded state is found it will send an snmptrap to your snmptrap collector.

#!/usr/bin/perl
#################################################################
# Script checks status of hardware raid on X86 hardware        #
# Supported OS's: Solaris x86, Redhat                #
# Supported Controller's: LSI & Adaptec                #
# Note: Requires mpt-status for LSI                #
# Note: Requires arccon for adaptec                #
# Note: Requires raidctl for Solaris x86            #
# Sends trap if degraded state                    #
#################################################################
use strict;
my $prefix = "bs_raid_chk";
my $servicechk = "unix_traps";
my $community = "asdpublic";
my $manager = "10.66.65.23";
my $raidctl = '/usr/sbin/raidctl';
my $mptstatus = '/usr/sbin/mpt-status';
my $arcconf = '/usr/StorMan/arcconf';
my (@components,@command);
my ($num,$status,$sendtrap,$volume);
### If Solaris System use this check ###
if (`uname -a` =~ /SunOS/) {
    if ( -e $raidctl) {
        @command = `$raidctl -S`;
        foreach (@command) {
            chomp();
            @components = split();
            $num = $components[1] + 2;
            $status = "$components[$#components] - Controller: $components[0], RAID: $components[$num], Number of Disks: $components[1]\n";
            if ($_ =~ /DEGRADED/) {
                system "/bin/rm /tmp/$prefix.* >/dev/null 2>&1";
                system "/bin/touch /tmp/$prefix.CRITICAL";
                system "/usr/sfw/bin/snmptrap -v 2c -c $community $manager '' .1.3.6.1.4.1.11.2.17.1.0.1005 .1.3.6.1.4.1.11.2.17 s \"$servicechk\" .1.3.6.1.4.1.11.2.17 s \"$status\"";
                        exit;
            } elsif ($_ =~ /SYNC/) {
                system "/bin/rm /tmp/$prefix.* >/dev/null 2>&1";
                                system "/bin/touch /tmp/$prefix.WARNING";
                system "/usr/sfw/bin/snmptrap -v 2c -c $community $manager '' .1.3.6.1.4.1.11.2.17.1.0.1006 .1.3.6.1.4.1.11.2.17 s \"$servicechk\" .1.3.6.1.4.1.11.2.17 s \"$status\"";
                        exit;
            } elsif ($_ =~ /OPTIMAL/) {
                if ( !-e "/tmp/$prefix.OK" ) {
                    system "/bin/rm /tmp/$prefix.* >/dev/null 2>&1";
                    system "/bin/touch /tmp/$prefix.OK";
                    system "/usr/sfw/bin/snmptrap -v 2c -c $community $manager '' .1.3.6.1.4.1.11.2.17.1.0.1007 .1.3.6.1.4.1.11.2.17 s \"$servicechk\" .1.3.6.1.4.1.11.2.17 s \"$status\"";
                }
            }
        }
    }
}
### If Linux system use this check  ###
if (`uname -a` =~ /Linux/) {
    # If system has LSI controller, then mptstatus should be installed
    if (-e $mptstatus) {
        my $modstatus = `/sbin/lsmod |grep mptctl|wc -l`;
        chomp($modstatus);
        if ($modstatus eq "0") {
            my $modload = `/sbin/modprobe mptctl`;
            $modstatus = `/sbin/lsmod |grep mptctl|wc -l`;
            chomp($modstatus);
            if ($modstatus eq "0") { print "ABORT: Failed to load mptctl module.\n";exit;}
        }
        my $controller = `$mptstatus -p -s|grep Found`;
        chomp($controller);
        my ($id,$junk) = split(/,/,$controller);
        $id =~ s/Found SCSI id=//g;
        @command = `$mptstatus -i $id -s`;
        $status="";
        foreach (@command) {
            chomp();
            $status = "$status $_";   
        }
        $status = "$status";
        #print "$status\n";
        foreach (@command) {
            chomp();
            if ( $_ =~ /DEGRADED/ ) {
                system "/bin/rm /tmp/$prefix.* >/dev/null 2>&1";
                system "/bin/touch /tmp/$prefix.CRITICAL";   
                system "/usr/bin/snmptrap -v 2c -c $community $manager '' .1.3.6.1.4.1.11.2.17.1.0.1005 .1.3.6.1.4.1.11.2.17 s \"$servicechk\" .1.3.6.1.4.1.11.2.17 s \"$status\"";
                exit;
            } elsif ($_ =~ /SYNC/ ) {
                system "/bin/rm /tmp/$prefix.* >/dev/null 2>&1";
                system "/bin/touch /tmp/$prefix.WARNING";
                system "/usr/bin/snmptrap -v 2c -c $community $manager '' .1.3.6.1.4.1.11.2.17.1.0.1006 .1.3.6.1.4.1.11.2.17 s \"$servicechk\" .1.3.6.1.4.1.11.2.17 s \"$status\"";
                exit;
            } elsif ($_ =~ /OPTIMAL/ ) {
                if ( !-e "/tmp/$prefix.OK" ) {
                    system "/bin/rm /tmp/$prefix.* >/dev/null 2>&1";
                    system "/bin/touch /tmp/$prefix.OK";
                    system "/usr/bin/snmptrap -v 2c -c $community $manager '' .1.3.6.1.4.1.11.2.17.1.0.1007 .1.3.6.1.4.1.11.2.17 s \"$servicechk\" .1.3.6.1.4.1.11.2.17 s \"$status\"";
                }
            }
        }
    }
    # if system has Adaptec controller then arcconf should be installed
    if ( -e $arcconf ) {
        @command = `$arcconf getconfig 1|grep Status|grep :`;
        foreach (@command) {
            if (( $_ =~ /Controller Status/ ) && ($_ !~ /Optimal/ )) {
                $status = "Controller not optimal";
                system "/bin/rm /tmp/$prefix.* >/dev/null 2>&1";
                system "/bin/touch /tmp/$prefix.CRITICAL";
                system "/usr/bin/snmptrap -v 2c -c $community $manager '' .1.3.6.1.4.1.11.2.17.1.0.1005 .1.3.6.1.4.1.11.2.17 s \"$servicechk\" .1.3.6.1.4.1.11.2.17 s \"$status\"";
                exit;
            }
            if (( $_ =~ /  Status  / ) && ($_ !~ /Optimal/ )) {
                $status = "Battery not optimal";
                system "/bin/rm /tmp/$prefix.* >/dev/null 2>&1";
                system "/bin/touch /tmp/$prefix.WARNING";
                system "/usr/bin/snmptrap -v 2c -c $community $manager '' .1.3.6.1.4.1.11.2.17.1.0.1006 .1.3.6.1.4.1.11.2.17 s \"$servicechk\" .1.3.6.1.4.1.11.2.17 s \"$status\"";
                exit;
            }
            if (( $_ =~ /Status of logical device/) && ($_ !~ /Optimal/ )) {
                $status = "Logical HW RAID Volume not optimal";
                system "/bin/rm /tmp/$prefix.* >/dev/null 2>&1";
                system "/bin/touch /tmp/$prefix.CRITICAL";
                system "/usr/bin/snmptrap -v 2c -c $community $manager '' .1.3.6.1.4.1.11.2.17.1.0.1005 .1.3.6.1.4.1.11.2.17 s \"$servicechk\" .1.3.6.1.4.1.11.2.17 s \"$status\"";
                exit;
            }
            if ( $_ =~ /Optimal/ ) {
                if ( !-e "/tmp/$prefix.OK" ) {
                    $status = "Hardware RAID - OK";
                    system "/bin/rm /tmp/$prefix.* >/dev/null 2>&1";
                    system "/bin/touch /tmp/$prefix.OK";
                    system "/usr/bin/snmptrap -v 2c -c $community $manager '' .1.3.6.1.4.1.11.2.17.1.0.1007 .1.3.6.1.4.1.11.2.17 s \"$servicechk\" .1.3.6.1.4.1.11.2.17 s \"$status\"";
                }
            }
        }
    }   
}
exit; 

Replacing Failed Disk in Linux Software Raid

When I worked at the University of Minnesota we used IBM x86 servers and since well it was the University we didn't buy a lot of systems with hardware raid.   This meant that software raid in Linux was the rage.   Unfortunately the team I worked with never really documented the process very well and so the following write up takes a practical example and illustrates the steps.

Overview:

 1) Determine which disk has failed.

 2) Remove failed disk from metadevice using mdadm. 

 3) Physically replace disk. 

 4) Partition new disk using sfdisk. 

 5) Add new disk back into raid metadevice using mdadm. 

 6) Confirm array is rebuilding by "cat /proc/mdstat". 

 Example: 

 1) Below, md3 and md4 have a failed device sdc, which contains slices sdc1 and sdc2.

#cat /proc/mdstat
Personalities : [raid1] [raid5]
read_ahead 1024 sectors
Event: 5
md2 : active raid1 sdb1[1] sda1[0]
80192 blocks [2/2] [UU]
resync=DELAYED
md0 : active raid1 sdb2[1] sda2[0]
2096384 blocks [2/2] [UU]
resync=DELAYED
md1 : active raid1 sdb3[1] sda3[0]
33366912 blocks [2/2] [UU]
[===>.................]  resync = 16.5% (5511296/33366912) finish=46.2min speed=10027K/sec
md3 : active raid5 sde1[2] sdd1[1]
35342720 blocks level 5, 64k chunk, algorithm 0 [3/2] [_UU]
md4 : active raid5 sde2[2] sdd2[1]
35744384 blocks level 5, 64k chunk, algorithm 0 [3/2] [_UU]

2) We need to use mdadm to remove failed slices from array. Note this is not always needed, but we will show example for practicle purposes.

#mdadm -r /dev/md3 /dev/sdc1
#mdadm -r /dev/md4 /dev/sdc2

3) Physically remove the disk(s) from the system. 4) Partition the new disk, using a partition table from another member(device) of the array. This example will use /dev/sde and we will dump partition table out to a file and then read it into the new device using sfdisk.

#sfdisk -d /dev/sde>/tmp/partition.out
#sfdisk /dev/sdc

5) Add the device slices back into corresponding raid metadevice using mdadm.

#mdadm -a /dev/md3 /dev/sdc1
#mdadm -a /dev/md4 /dev/sdc2

6) Cat /proc/mdstat for results.

#cat /proc/mdstat
Personalities : [raid1] [raid5]
read_ahead 1024 sectors
Event: 8
md2 : active raid1 sdb1[1] sda1[0]
80192 blocks [2/2] [UU]
resync=DELAYED
md0 : active raid1 sdb2[1] sda2[0]
2096384 blocks [2/2] [UU]
resync=DELAYED
md1 : active raid1 sdb3[1] sda3[0]
33366912 blocks [2/2] [UU]
[==========>..........]  resync = 52.8% (17633280/33366912) finish=25.6min speed=10228K/sec
md3 : active raid5 sdc1[3] sde1[2] sdd1[1]
35342720 blocks level 5, 64k chunk, algorithm 0 [3/2] [_UU]
[============>........]  recovery = 60.9% (10776268/17671360) finish=11.2min speed=10233K/sec
md4 : active raid5 sdc2[3] sde2[2] sdd2[1]
35744384 blocks level 5, 64k chunk, algorithm 0 [3/2] [_UU]

Note: md4 will not start rebuilding until md3 is complete, since both raids contain slices from the same physical disk.