Expire List of Tapes in Netbackup

When working in a large Netbackup environment, there often comes a time when you need to expire a large amount of tapes all at once.

This was exactly the scenario that happened to me when a company I consulted at changed their retention periods.  The change of retention periods from 1 year to 1 month meant they wanted to free up all the tapes that contained images that were more then a month old.

The solution to the problem was a script that basically does the following:

1) Reads in a list of media id's from a file.
2) Determine which media server the media id is assigned.
3) Expire the media id from the Netbackup catalog.

The script is here:

#!/usr/bin/perl
$master="hostname of master server";
open DATA, "cat /home/schmaubj/media.list|";
while () {
 $mediaid = $_;
 chomp($mediaid);
 open DATA2, "/usr/openv/netbackup/bin/admincmd/bpmedialist -U -m $mediaid|";
 while () {
  $line = $_;
  chomp($line);
  if ($line =~ /Server Host/) {
   ($junk,$mhost) = split(/=/,$line); 
   chomp($mhost);
    $mhost =~ s/ *$//;  
          $mhost =~ s/^ *//; 
  }
 }
 close (DATA2);
 print "Media ID: $mediaid\n";
 print "Media Server Host: $mhost\n";
 print "Expiring now...\n";
 $expire=`/usr/openv/netbackup/bin/admincmd/bpexpdate -force -d 0 -m $mediaid -host $mhost -M $master`;
}
close (DATA);

Netbackup Auto Discovery of VM's

When I first started to work with Netbackup and VMWare VCB backups, the biggest complaint I often found was that Netbackup could not auto discover new VM's and automatically insert them into a given policy. Along with that I often found that if I was running more then one VCB backup against my Netbackup proxy server, and one or more of those VM's resided on the same VMWare datastore, I would often get a snapshot error, as both tried to compete to get a snapshot on the datastore. These issues made Netbackup VCB backups problematic and management intensive for the large environment of 50+ VM's.

Pondering this issue, I came up with an elegant yet useful solution. The idea was to create multiple policies, one for each datastore in my VMWare cluster. The policy would be configured to run one client at a time, and all the policies would run at the same time. In the end you would end up with VCB backup jobs running against the proxy server, but they would not be on the datastore since each policy was datastore specific. This eliminated the snapshot errors and also allowed for me to have the policies auto updated.

Like I said, I was able to have my policies auto updated with new VM clients without any intervention. This was made possible by writing a script that used the VMWare Perl API's and made calls into both VMWare and then Netbackup. Here is how the setup works:

1) Create a Netbackup policy for each datastore in your VMWare environment. Make sure the policies all have a naming convention. Example: NB_VCB_datastore1, NB_VCB_datastore2, NB_VCB_datastore3, etc...
2) Use the script below and run from the Unix/Linux master server or any Unix/Linux server that has the Netbackup admincmd directories installed and is allowed access to the Netbackup master server. Make sure you have the VMWare Perl API libraries installed with Perl on the master server.

Usage: autovcb -v (vc server) -u (username) -p (password) -d (datacenter in vc) -os (Windows|Solaris|Linux|Suse|Redhat) -pp (netbackup policy prefix) -m (master server) -dsp (datastore name|ALL) [-pre (datastore match prefix)] [-update]

Note: The Netbackup policy prefix is the naming convention value you used before the datastore in the policy name. In my example it is "NB_VCB_". Also, be aware that forward and reverse DNS must be functional on the VM hosts to ensure proper inclusion in the policy.

The script allows you to only match on some datastore or all datastores. You can do a full discovery and populate or just an update of the policies. You can also chose to only add clients of a specific OS.

This script was tested in a Netbackup 6.5.4 and VMWare ESX 3.5 environment.

The script is here:

#!/usr/bin/perl
#################################################################################
# Script that autodiscovers VM's and places the client name into prexisting #
# policies based on the datastore names discovered    #
# (C) Benjamin Schmaus May, 2009      #
################################################################################# 
use strict;
use warnings;
use Socket;
use VMware::VIRuntime;
use VMware::VILib;
use VMware::VIM25Stub;
use Getopt::Long;

### Main Logic ###
my $bpplclients = '/usr/openv/netbackup/bin/admincmd/bpplclients';
my ($help,$prefix,$vcserver,$username,$password,$dc,$os,$polpr,$datastorepol,$update,$master);
my ($datastorepoltmp,$listds,$esxhost,$vmname,$adhostname,$ipaddress,$toolstat,$guestos,$datastore,$hostname,$junk,$junk2,$junk3,$client,$datastorefix,$polname,$gueststate);
my @dslist = ();
options();
print "update is set to: $update\n";
my $policy = "$polpr$datastorepol";
my $url = "https://$vcserver/sdk/vimService";
Vim::login(service_url => $url, user_name => $username, password => $password);
my $datacenter_views = Vim::find_entity_views(view_type => 'Datacenter',filter => { name => $dc });
my $dcds = Vim::find_entity_view(view_type => "Datacenter", filter => {'name' => "Fridley"} );
my $ds = $dcds->datastore;

if ($datastorepol eq "ALL") {
 getlistds();
 print "\n";
}
if ($update eq "0") {
 polrmclients();
 print "\n";
}

if ($datastorepol eq "ALL") {
 print "Discovering new clients...\n";
 # go through all policies and add new clients for all datastores
 foreach (@dslist) {
  $datastorepoltmp = $_; 
  print "\tDatastore: $datastorepoltmp\n";
  print "\tPolicy: $polpr$datastorepoltmp\n";
  polgetclients();
  print "\t\n";
 }
} else {
 print "Discovering new clients...\n";
 # go through all policies and add new clients for given datastore
 $datastorepoltmp = $datastorepol;
 print "\tCurrent Datastore: $datastorepoltmp\n";
 print "\tCurrent Policy: $$polpr$datastorepoltmp\n";
 polgetclients();
 print "\t\n";
}
print "\n\n";
Vim::logout();
exit;


sub polgetclients {
 foreach (@$datacenter_views) {
  my $datacenter = $_->name;
  my $host_views = Vim::find_entity_views(view_type => 'HostSystem',begin_entity => $_ );
  foreach (@$host_views) {
   $esxhost = $_->name;
   #print "\tESX Host Server: $esxhost\n";
   my $vm_view = Vim::find_entity_views(view_type => 'VirtualMachine',begin_entity => $_ , filter => { 'guest.guestState' => 'running' });
   foreach (@$vm_view) {
    $vmname = $_->name;
    $adhostname = $_->summary->guest->hostName;
    $ipaddress = $_->summary->guest->ipAddress;
    $toolstat = $_->summary->guest->toolsStatus->val;
    $guestos = $_->summary->guest->guestFullName;
    $datastore = $_->summary->config->vmPathName;
    $gueststate = $_->guest->guestState;
    ($datastorefix,$junk) = split(/\] /,$datastore);
    $datastorefix =~ s/\[//g;
    $datastorefix =~ s/ /_/g;
    if ($ipaddress) {
     $hostname = gethostbyaddr(inet_aton($ipaddress), AF_INET);
     #print "$ipaddress $hostname\n";
     #if (!$hostname) { $hostname = '---'; }
     #$hostname =~ s/.asd.udlp.com/.mpls.udlp.com/g;
     #if (($hostname !~ /.mpls.udlp.com/) && ($hostname !~ /udlp.com/)) {
     # $hostname = "$hostname.mpls.udlp.com";
     #}
    }
    if (!$hostname) { $hostname = '---'; }
    $policy = "$polpr$datastorefix";
    if ($datastorefix eq $datastorepoltmp) {
     if ($toolstat =~ /toolsOk/) {
      if ($guestos =~ /$os/) {
       if ($hostname ne "---" && $hostname ne "localhost") {
        my $addtopolicy = `$bpplclients $policy -M $master -add $hostname VMware Virtual_Machine > /dev/null 2>&1`;
        if ($? eq "0") {
         print "\t\tAdded $hostname\n";
        } else {
         print "\t\tSkipped $hostname: Already in policy\n";
        }
       } else {
        print "\t\tSkipped $vmname: Reverse DNS does not exist\n";
       }
      } else {
       print "\t\tSkipped $vmname: OS does not match guestos\n";
      }
     } else {
      print "\t\tSkipped $vmname: Tools not okay - $toolstat\n";
     }
    }
   }
  }
 }
}

sub polrmclients {
 if ($datastorepol eq "ALL") {
  foreach (@dslist) {
   $policy = "$polpr$_";
   print "Removing clients from $policy...\n";
   my @polrm = `$bpplclients $policy -noheader`;
   foreach (@polrm) {
    ($junk,$junk2,$client) = split(/ /,$_);
    chomp($client);
    my $remove = `$bpplclients $policy -M $master -delete $client > /dev/null 2>&1`;
    if ($? eq "0") {
                                 print "\t\tRemoved $client\n";
                                } else {
                                 print "\t\tSkipped $client: Not in policy\n";
    }
   }
  }
 } else { 
  $policy = "$polpr$datastorepol";
  my @polrm = `$bpplclients $policy -noheader`;
  print "Removing clients from $policy...\n";
  foreach (@polrm) {
   ($junk,$junk2,$client) = split(/ /,$_);
   chomp($client);
   my $remove = `$bpplclients $policy -M $master -delete $client > /dev/null 2>&1`;
   if ($? eq "0") {
                         print "\t\tRemoved $client\n";
                        } else {
                         print "\t\tSkipped $client: Not in policy\n";
                        }
  }
 }
}

sub getlistds {
 print "Getting list of datastores...\n";
 my $counter = 0;
 foreach(@$ds) {
         my $ds_ref = Vim::get_view(mo_ref => $_);
         my $ds = $ds_ref->info->name;
         if (($ds =~ /$prefix/) && ($prefix ne "---")) {
                 $dslist[$counter] = $ds_ref->info->name;
                 $dslist[$counter] =~ s/ /_/g;
   $dslist[$counter] =~ s/\(|\)//g;
   print "\tFound datastore: $dslist[$counter]\n";
                 ++$counter;
         } elsif  ($prefix eq "---") {
                 $dslist[$counter] = $ds_ref->info->name;
                 $dslist[$counter] =~ s/ /_/g;
   $dslist[$counter] =~ s/\(|\)//g;
   print "\tFound datastore: $dslist[$counter]\n";
                 ++$counter;
         }
 }
}

sub options {
        $vcserver="";$username="";$password="";$dc="";$os="";$polpr="";$master="";$update="0";$prefix="";$datastorepol="";
        GetOptions ('h|help'=>\$help,'v|vcserver=s'=>\$vcserver,'u|username=s'=>\$username,'p|password=s'=>\$password,'d|datacenter=s'=>\$dc,'os=s'=>\$os,'pp|polpr=s'=>\$polpr,'m|master=s'=>\$master,'dsp=s'=>\$datastorepol,'pre|prefix=s'=>\$prefix,'update'=>\$update);
        if ($help) {
                print "Usage: autovcb -v  -u  -p  -d  -os  -pp  -m  -dsp <(datastore name|ALL)> [-pre ] [-update]\n";
                exit;
        }
        if (($vcserver eq "") || ($username eq "") || ($password eq "") || ($dc eq "") || ($os eq "") || ($polpr eq "") || ($master eq "") || ($datastorepol eq "")) {
  print "Missing required parameters - Type -help for options\n";
                exit;
        }
 if (($prefix) && ($datastorepol ne "ALL")) {
  print "Cannot use prefix option when datastore is not set to ALL - Type -help for options\n";
  exit;
 }

        if (($os ne "Windows") && ($os ne "Solaris") && ($os ne "Linux") && ($os ne "Suse") && ($os ne "Redhat")) {
  print "Incorrect OS specified - Type -help for options\n";
                exit;
        }
}

VMWare Tool Check

If you have ever worked in a large VMWare environment, you know what a pain it can be to determine if the VMWare tools are up to date on a VMWare host. That is why I wrote the following Perl script. The script enumerates through the VMWare hosts and tells you if the VMWare tools are up to date or not.


The link to the script is here:

http://home.comcast.net/~schmaustech/source/vmtoolchk.zip

Perl & VMWare Snapshots & Netbackup

In VmWare, Virtual Center has the ability for you to take snapshots. These might come in handy before you install a new application, perform a OS upgrade or make any other significant change to the VM.

Netbackup also creates a snapshot when you do a VCB backup or incremental backup against a VM. Sometimes these snapshots are left behind when a backup of the VM fails. The result is that you have to go into Virtual Center, find the VM and remove the snapshot. This can be time consuming and lead me to develop a tool in Perl as a solution.

The Perl solution I came up with was inspired by the sample snapshot Perl script that comes with the VMWare Perl API SDK. This script however was not specific for my needs. So I came up with a script that would delete snapshots based on a pattern match of the name of the snapshot.

Basically the script will connect to a Virtual Center server and enumerate through the hosts associated with that Virtual Center. While enumerating through the hosts, it checks for snapshots and then compares the pattern entered to the name of the snapshot. If they match it will remove it. In the case of a Netbackup VCB backup snapshot, the names always start with "VCB_". Thus if you use "VCB_" as your pattern you can remove all snapshots on all your hosts. You can extend this to all snapshots if you use a naming convention when you create snaps.

The script is here:

#!/usr/bin/perl
#################################################################################
# Script that autodiscovers VM's and places the client name into prexisting     #
# policies based on the datastore names discovered                              #
# (C) Benjamin Schmaus May, 2009                                                #
#################################################################################
use strict;
use warnings;
use Socket;
use VMware::VIRuntime;
use VMware::VILib;
use VMware::VIM25Stub;
use Getopt::Long;
my ($help,$vcserver,$username,$password,$dc,$removesnapshot,$children,$snapshotname);
$children="0";
options();
my $url = "https://$vcserver/sdk/vimService";
Vim::login(service_url => $url, user_name => $username, password => $password);
my $datacenter_views = Vim::find_entity_views(view_type => 'Datacenter',filter => { name => $dc });
list_snapshot();
exit;

sub list_snapshot {
 my $datacenter_views = Vim::find_entity_views(view_type => 'Datacenter',filter => { name => $dc });
    foreach (@$datacenter_views) {
  my $datacenter = $_->name;
  my $host_views = Vim::find_entity_views(view_type => 'HostSystem',begin_entity => $_ );
  foreach (@$host_views) {
   my $esxhost = $_->name;
   my $vm_view = Vim::find_entity_views(view_type => 'VirtualMachine',begin_entity => $_ , filter => { 'guest.guestState' => 'running' });
   foreach (@$vm_view) {
          my $mor_host = $_->runtime->host;
          my $hostname = Vim::get_view(mo_ref => $mor_host)->name;
    my $ref = undef;
    my $nRefs = 0;
          my $count = 0;
          my $snapshots = $_->snapshot;
          if(defined $snapshots) {
             Util::trace(0,"\nSnapshots for Virtual Machine ".$_->name . " under host $hostname\n");
             printf "\n%-47s%-16s %s %s\n", "Name", "Date","State", "Quiesced";
             print_tree ($_->snapshot->currentSnapshot, " ", $_->snapshot->rootSnapshotList);
     if(defined $_->snapshot) {
                                         ($ref, $nRefs) = find_snapshot_name ($_->snapshot->rootSnapshotList, $snapshotname);
                                 }
     if ($snapshotname =~ /$removesnapshot/) {
      print "Snapshot name: $snapshotname matches for delete\n";

            if (defined $ref && $nRefs == 1) {
               my $snapshot = Vim::get_view (mo_ref =>$ref->snapshot);
               eval {
                   $snapshot->RemoveSnapshot (removeChildren => $children);
                    Util::trace(0, "\nRemove Snapshot ". $snapshotname . " For Virtual Machine ". $_->name . " under host $hostname" ." completed sucessfully\n");
       };
               if ($@) {
                   if (ref($@) eq 'SoapFault') {
                       if(ref($@->detail) eq 'InvalidState') {
                          Util::trace(0,"\nOperation cannot be performed in the current state of the virtual machine");
                       } elsif(ref($@->detail) eq 'HostNotConnected') {
                          Util::trace(0,"\nhost not connected.");
                       } else {
                          Util::trace(0, "\nFault: " . $@ . "\n\n");
                       }
                   } else {
                       Util::trace(0, "\nFault: " . $@ . "\n\n");
                   }
               }
            } else {
               if ($nRefs > 1) {
                   Util::trace(0,"\nMore than one snapshot exits with name" ." $snapshotname in Virtual Machine ". $_->name ." under host ". $hostname ."\n");
               }
               if($nRefs == 0 ) {
                   Util::trace(0,"\nSnapshot Not Found with name" ." $snapshotname in Virtual Machine ". $_->name ." under host ". $hostname ."\n");
               }
      }
     }
          #} else {
            # Util::trace(0,"\nNo Snapshot of Virtual Machine ".$_->name ." exists under host $hostname\n");
          }
   }
  }
 }
}

sub options {
        $vcserver="";$username="";$password="";$dc="";$removesnapshot="";
        GetOptions ('h|help'=>\$help,'v|vcserver=s'=>\$vcserver,'u|username=s'=>\$username,'p|password=s'=>\$password,,'d|datacenter=s'=>\$dc,'sm|snapmatch=s'=>\$removesnapshot);
        if ($help) {
                print "Usage: snapper.pl -v  -u  -p  -d  -sm \n";
                exit;
        }
        if (($vcserver eq "") || ($username eq "") || ($password eq "") || ($dc eq "") || ($removesnapshot eq "")) {
                print "Missing required parameters - Type -help for options\n";
                exit;
        }

}

sub print_tree {
 my ($ref, $str, $tree) = @_;
    my $head = " ";
    foreach my $node (@$tree) {
        $head = ($ref->value eq $node->snapshot->value) ? " " : " " if (defined $ref);
        my $quiesced = ($node->quiesced) ? "Y" : "N";
        $snapshotname = $node->name;
        printf "%s%-48.48s%16.16s %s %s\n", $head, $str.$node->name,
              $node->createTime, $node->state->val, $quiesced;
        print_tree ($ref, $str . " ", $node->childSnapshotList);
    }
    return;
}

sub find_snapshot_name {
 my ($tree, $name) = @_;
    my $ref = undef;
    my $count = 0;
    foreach my $node (@$tree) {
        if ($node->name eq $name) {
           $ref = $node;
           $count++;
        }
        my ($subRef, $subCount) = find_snapshot_name($node->childSnapshotList, $name);
        $count = $count + $subCount;
        $ref = $subRef if ($subCount);
    }
    return ($ref, $count);
}

Netapp Aggregate/Volume Report

Getting information from the Netapp is easy with the web interface and the command line interface. However sometimes I want information in a Excel format and I want it delivered to me in a automated easily readable format.

This was the case when I created the following Perl based script that gathers all the volume/aggregate usage information from a Netapp filer or group of Netapp filers and sends that information via SMTP in a easy to read Excel format.

The link to the script is here:

http://home.comcast.net/~schmaustech/source/netapp-aggr.zip

Netapp Coverage Report for Symantec Netbackup

Anyone who has used Symantec Netbackup should be familiar with the coverage report script provided in the goodies directory. The script provides the ability to see what filesystems are being backed up on the clients in your Netbackup policies. The problem with the script is that it relies on accessing clients that have the Netbackup client installed on them. This works great except when it comes to NDMP backup policies for Netapps.

The reality is that the coverage report supplied does not provide details on what volume paths are being covered and which ones are being missed when using Netbackup to backup a Netapp filer. However, after being approached to write a solution to the issue, I came up with a script that provides that missing coverage information.

The script I wrote is Perl based and at this point needs to be run from a Unix/Linux master server (may work on Windows but was never tested). The script, when edited and provided with the proper parameters, will go out and gather the path information from one or more Netapp filers and then compare those hosts and paths to what you have in your Netbackup polices for NDMP backups. The results are then emailed off in an Excel formatted spreadsheet.

The script is here:

#!/usr/bin/perl
#########################################################
# Netapp Netbackup Coverage Report                      #
#########################################################
use strict;
use Net::SSH::Perl;
use Spreadsheet::WriteExcel;
my @hosts = ("netapp1","netapp2");
my (@output);my $hcount = 0;my $user="root";my $password="password";
my (@netpaths,@tmpnetpaths,$host,$tmparray,$output,$name,$path,$comment,$coverage);
my $subject = "Netapp_Backup_Coverage_Report";
my @emails = ('benjamin.schmaus\@schmaustech.com');
my (%saw,$scon,$errors,$exit,$netpaths,@tmparray);
my ($include,$policyname,$junk,@ppaths,$ppaths,$loop);
my ($emails,$mailit,$message);
my $num="2";

### Setup Excel Worksheet ###
my $workbook  = Spreadsheet::WriteExcel->new('/tmp/ncoverage.xls');
my $worksheet = $workbook->add_worksheet();
my $format = $workbook->add_format();
my $header = $workbook->add_format();
create_excel();

foreach $host (@hosts) {
 print "Gathering paths from $host...\n";
 $scon = Net::SSH::Perl->new ("$host",protocol => 2);
 $scon->login("$user","$password");
 ($output[$hcount],$errors,$exit) = $scon->cmd("cifs shares;logout telnet");
 $hcount++;
}
foreach $output (@output) {
 @tmparray = split(/\n/,$output);
 foreach $tmparray (@tmparray) {
  if ($tmparray =~ /\/vol/) {
   ($name,$path,$comment) = split(' ',$tmparray);
   $path =~ tr/[A-Z]/[a-z]/;
   push(@tmpnetpaths,$path);
  }
 }
}
print "Sorting paths from Netapps...\n";
undef %saw;
@saw{@tmpnetpaths} = ();
@netpaths = sort keys %saw;
foreach $host (@hosts) {
 print "Gathering backup selections from Netbackup for $host...\n";
        open DATA, "/usr/openv/netbackup/bin/admincmd/bppllist -allpolicies -U -byclient $host|";
        while () {
                chomp();
  $_ =~ s/\s//g;
  if ($_ =~ /PolicyName/) {
                        ($junk,$policyname) = split(/:/);
  }
  if ($_ =~ /\/vol\//) {
                 $_ =~ s/\s//g;
   $_ =~ s/Include://g;
   $_ =~ s/\/*$//g;
   $_ =~ tr/[A-Z]/[a-z]/;
   push (@ppaths,"$policyname:$_");
  }
        }
        close (DATA);
}
#### Reconcile time ####
print "Reconcile paths...\n";
foreach $netpaths (@netpaths) {
 $path = $netpaths;
 $loop ="0";
 foreach $ppaths (@ppaths) { 
  ($policyname,$include) = split(/:/,$ppaths);
  if ($netpaths =~ /$include/) {
   $coverage="COVERED";
   cell();
   $loop ="1";
  }
 }
 if ($loop eq "0") {
  $coverage="UNCOVERED";
  $policyname="NONE";
  cell();
 }
}

$workbook->close();


### Mail Off Results ###
print "Mailing off results.\n";
mailit();
exit;

### Setup Excel Format Subroutine ###
sub create_excel {
        $format->set_bold();
        $format->set_size(16);
        $format->set_align('center');
        $header->set_bold();
        $header->set_align('center');
        $worksheet->set_column(0, 0, 40);
        $worksheet->set_column(1, 1, 20);
        $worksheet->set_column(2, 2, 20);
        $worksheet->write(1, 0,  'Netapp Path', $header);
        $worksheet->write(1, 1,  'Netbackup Policy', $header);
        $worksheet->write(1, 2,  'Status', $header);
        $worksheet->merge_range('A1:F1','Netapp Backup Coverage Report',$format);
}

### Mail Subroutine ###
sub mailit {
        $message = `echo "'Netapp Backup Coverage Report">/tmp/ncr-body.txt`;
        $message = `echo "">>/tmp/ncr-body.txt`;
        $message = `/usr/bin/uuencode /tmp/ncoverage.xls ncoverage.xls > /tmp/ncr-attachment.txt`;
        $message = `cat /tmp/ncr-body.txt /tmp/ncr-attachment.txt > /tmp/ncr.txt`;
        foreach $emails (@emails) {
                $mailit = `/usr/bin/mailx -s $subject $emails < /tmp/ncr.txt`;
        }
}

sub cell {
 #print "NETPATH: $path\n";
 $worksheet->write($num,0,$path);
        $worksheet->write($num,1,$policyname);
        $worksheet->write($num,2,$coverage);
        $num = $num + 1;
}