Monitoring Sensors and Taking Action

I recently wrote a blog around using Microshift to run my Zigbee2MQTT workload. This blog described all the details on how to deploy Microshift and then deploy the components inside of Microshift to enable some home automation. Of course with Zigbee2MQTT there is an intuitive web interface to interact with the smart devices. However I wanted to take another approach that felt more realistic when it comes to edge use cases. I felt that in a industrial scenario there would be some code that would most likely subscribed and monitoring the MQTT queue. An action would be performed when a certain event was observed and the action itself might publish something into the MQTT queue. The rest of this blog will cover a simple scenario like I just described.

First we continue to use the same lab environment I used in my previous blog. The only difference here in the diagram below is we have now added a smart power outlet and a temperature/humidity sensor that can both be controlled remotely via the Zigbee protocol like all my other devices.

The Script

With my lab in place I decided I wanted ot write something in Perl. Some might think why use such an antiquated language like Perl and part of that is because I am old school. For my scenario I envisioned using the humidity sensor to detect when the humidity levels got too high. The threshold would then trigger an action on the event to turn on/off a dehumidifier plugged into the smart outlet. The basic process flow looks like the following diagram:

The script itself can take four different parameters:

• --hostname: hostname or IP address of MQTT host (required)
• --port: port for MQTT (optional but will default to 1883 if not provided)
• --threshold: humidity value that determines when action should be taken
• --help: prints the usage of script

The script itself is located here

When one runs the script without any flags the usage and an example will be displayed.

./mqtt-humidity.pl 
Usage:
      --hostname,-h   Hostname or IP address of MQTT host
      --port,-p       Port for MQTT (defaults to default 1883)
      --threshold,-t  Threshold for humidity (defaults to 60)
      --help,-h       Print this help

    Example:

    mqtt-humidity.pl -ho 10.43.26.170 -p 1883 -t 65

The Demonstration of Script

To demonstrate this script I went ahead and plugged in a light into my smart outlet which was in the off setting. I launched the script in a terminal window. Then I took the temperature/humidity sensor, cupped it in my hands and blew into my hands. The moisture in my breath is enough to temporarily raise the value. The script provides output so we can see the values changing and sure enough when I breathed into my hands with the sensor the value jumped to 81.37% which triggered the action event and turned on the light. I then set the sensor back on my desk and over the course of 5 minutes the value slowly receded. Once it dropped below the threshold value the light then turned back off. The output of my script run is below:

$ perl mqtt-humidity.pl -ho 10.43.26.170 -p 1883 -t 60
Temp C = 23.43 : Temp F = 74.174 : Humidity = 51.22
Temp C = 23.43 : Temp F = 74.174 : Humidity = 81.37 <-- Smart outlet turned on
Temp C = 23.43 : Temp F = 74.174 : Humidity = 84.37
Temp C = 23.43 : Temp F = 74.174 : Humidity = 82.37
Temp C = 23.43 : Temp F = 74.174 : Humidity = 80.28
Temp C = 23.43 : Temp F = 74.174 : Humidity = 78.79
Temp C = 23.63 : Temp F = 74.534 : Humidity = 78.79
Temp C = 23.63 : Temp F = 74.534 : Humidity = 73.21
Temp C = 23.63 : Temp F = 74.534 : Humidity = 74.34
Temp C = 23.63 : Temp F = 74.534 : Humidity = 72.91
Temp C = 23.63 : Temp F = 74.534 : Humidity = 71.65
Temp C = 23.63 : Temp F = 74.534 : Humidity = 70.55
Temp C = 23.63 : Temp F = 74.534 : Humidity = 69.15
Temp C = 23.63 : Temp F = 74.534 : Humidity = 67.93
Temp C = 23.63 : Temp F = 74.534 : Humidity = 66.57
Temp C = 23.63 : Temp F = 74.534 : Humidity = 64.87
Temp C = 23.63 : Temp F = 74.534 : Humidity = 63.28
Temp C = 23.63 : Temp F = 74.534 : Humidity = 62.08
Temp C = 23.63 : Temp F = 74.534 : Humidity = 60.71 
Temp C = 23.63 : Temp F = 74.534 : Humidity = 59.12 <-- Smart outlet turned off
Temp C = 23.63 : Temp F = 74.534 : Humidity = 57.72
Temp C = 23.63 : Temp F = 74.534 : Humidity = 56.7
Temp C = 23.63 : Temp F = 74.534 : Humidity = 55.6
Temp C = 23.63 : Temp F = 74.534 : Humidity = 54.23
Temp C = 23.63 : Temp F = 74.534 : Humidity = 53.21
Temp C = 23.63 : Temp F = 74.534 : Humidity = 52.14
Temp C = 23.63 : Temp F = 74.534 : Humidity = 51.05
Temp C = 23.63 : Temp F = 74.534 : Humidity = 50.04
Temp C = 23.22 : Temp F = 73.796 : Humidity = 50.04
Temp C = 23.22 : Temp F = 73.796 : Humidity = 50.04
^C

Now this was a very simple example but imagine the possibilities. For example what if this was a greenhouse that needed to keep the humidity and/or even the temperature at a certain range. If the device that reduces the humidity/temperature (dehumidifier -or- exhaust fan) in the greenhouse could take Zigbee commands directly and control the speed of operation we might be able to not only turn it on/off but also increase/decrease speed of operation. All of this ensures that whatever was growing in the greenhouse is not damaged and also ensures we are powering devices only when we need to have them powered. The bottom line is it saves businesses like the greenhouse operational costs when they are operating efficiently.

UCS Vmedia Policy with XML & Perl

In Cisco UCS there is a concept called a VMedia policy.  This policy allows you to designate a bootable ISO image from another server via HTTP protocol and then have it available as a boot device for a Cisco UCS blade in UCSM.   The following script is a rough framework in Perl that uses XML calls to configure such a policy.    This script could be elaborated on to take inputs for some of the various variables that I pre-populate as an example of how the framework would work.

#!/usr/bin/perl
use strict;
use LWP::UserAgent;
use HTTP::Request::Common;
my $ucs = "https:///nuova";       # This is the URL to your UCSM IP
my $username = "admin";           # Admin or other user to manager UCSM
my $password = "password";        # Password for user above
my $server = "ls-servername0001"; # Server name as defined in UCS convention
my $server2 = "servername0001";   # Server name from friendly view
my $policyname = "$server2";      # Policy name (derived from server name in this example)
my $mntename = "$server2";        # Mount name (derived from server name in this example)
my $type = "cdd";                 # Policy mount type (in this case CD-ROM)
my $image = "$server2.iso";       # Name of ISO image
my $imagepath = "/";              # Image path within the URL of remote host serving ISO
my $mountproto = "http";          # Protocol used to access ISO image
my $remotehost = "";              # Remote host IP serving ISO image
my $serverdn = "org-root/org-corp/$server";     # Server DN within UCSM  

###  Everything below remains constant###
### Get Cookie ###

my ($xmlout,@xmlout,$cookie);
my $login="";
my $userAgent = LWP::UserAgent->new;
my $response = $userAgent->request(POST $ucs, Content_Type => 'text/xml', Content => $login);

(@xmlout)= split(/\s+/,$response->content);

### Process Cookie ###

foreach $xmlout (@xmlout) {
        if ($xmlout =~ /outCookie/) {
                $cookie=$xmlout;
                $cookie =~ s/outCookie=\"|\"//g;
                print "$cookie\n";
        }
}

###Setup Vmedia Policy String###

my $crpolicy = "";

###Configure Mount Policy Within Vmedia Policy String###

my $mountentry = " ";

###Add Vmedia Policy String###

my $addvmedia = "  ";

###Execute Setup Vmedia Policy String###

$response = $userAgent->request(POST $ucs, Content_Type => 'text/xml', Content => $crpolicy);
(@xmlout)= split(/\s+/,$response->content);
printxml();
print "\n";

###Execute Configure Mount Policy Within Vmedia Policy String###

$response = $userAgent->request(POST $ucs, Content_Type => 'text/xml', Content => $mountentry);
(@xmlout)= split(/\s+/,$response->content);
printxml();
print "\n";

###Execute Add Vmedia Policy String###

$response = $userAgent->request(POST $ucs, Content_Type => 'text/xml', Content => $addvmedia);
(@xmlout)= split(/\s+/,$response->content);
printxml();
print "\n";
exit;

###Parse XML response###

sub printxml {
        foreach $xmlout (@xmlout) {
                if ($xmlout =~ /=/) {
                        $xmlout =~ s/\'|\"|\/|\>//g;
                        $xmlout =~ s/=/ = /g;
                        print "$xmlout\n";
                }
        }
}

Perl Script to Generate Logon.bat for SAMBA Users

The following script will generate a vanilla logon.bat file for SAMBA users. 

#!/usr/bin/perl
################################
# Usage: smb-logon-script      #
################################

$startpath="/data/smb-logon-scripts";
$endpath="/data/netlogon/scripts";
$smbhost = "sambahost.domain.com";


@alpha = ("g"..."t","v"..."z","aa"..."zz");
if (! defined $ARGV[0] ) {
        print " Usage: smb-logon-script \n";
        exit;
}
$username = $ARGV[0];
@group = `/usr/bin/getent group|/bin/grep $username |/bin/cut -d: -f1 -`;
$counter=0;
open FILE, ">$startpath/$username.bat.unix";
foreach $group (@group) {
        print FILE "net use $alpha[$counter]: \\\\$smbhost\\$group";
        $counter++;
}
close (FILE);
$convert = `/usr/bin/dos2unix < $startpath/$username.bat.unix > $endpath/$username.bat`;
exit;

Persistantly Bind Tape Devices in Solaris via Perl

The following script will look for fiber channel tape devices and then configure the devlinks.tab file with the appropriate information so the tape drives will persistently bind to the same device across reboots on a Solaris server.   This script was tested on Solaris 10.

#!/usr/bin/perl
use strict;
my($junk,$path,$devices,$dev,$file);
my(@devices,@file);
my $date = `date +%m%d%Y`;
$file = `/usr/bin/cp /etc/devlink.tab /etc/devlink.tab.$date`;
@file = `cat /etc/devlink.tab`;
@file = grep !/type=ddi_byte:tape/, @file;
open (FILE,">/etc/devlink.tab.new");
print FILE @file;
close (FILE);
 
@devices = `ls -l /dev/rmt/*cbn|awk {'print \$9 \$11'}`;
open (FILE,">>/etc/devlink.tab.new");
foreach $devices (@devices) {
        chomp($devices);
        ($dev,$path) = split(/\.\.\/\.\./,$devices);
        $dev =~ s/cbn//g;
        $dev =~ s/\/dev\/rmt\///g;
        $path =~ s/:cbn//g;
        ($junk,$path) = split(/st\@/,$path);
        print FILE "type=ddi_byte:tape;addr=$path;\trmt/$dev\\M0\n";
}
close (FILE);
$file = `/usr/bin/mv /etc/devlink.tab.new /etc/devlink.tab`;
exit;

Cleanup Shared Memory Segments Solaris

If you have ever used an application in Solaris that uses shared memory and that application has a tendency to not cleanup those memory segments properly on shutdown (SAP comes to mind)  then this little Perl script is what you have been waiting for.

All this script does is take certain field output from the ipcs command and then iterate through it to determine if the memory is still actively in use by a process or if it it can safely be purged.   I recommended testing this out with the $memclean line commented out to gain a good understanding before you remove the comment and allow the cleanup (#$memclean = `ipcrm -m $shmem`;).  This script was tested on Solaris 10.

#!/usr/bin/perl
@sms = `ipcs -pm|grep "^m"|awk {'print \$2":"\$7":"\$8'}`;
foreach $sms (@sms) {
        chomp($sms);
       ($shmem,$cpid,$lpid) = split(/:/,$sms);
        $cpids=` ps -ef|grep $cpid|grep -v grep >/dev/null 2>&1;echo \$?`;
        $lpids=` ps -ef|grep $lpid|grep -v grep >/dev/null 2>&1;echo \$?`;
        chomp($cpids,$lpids);
        if (($cpids eq "1") && ($lpids eq "1")) {
                $message = "Memory can be reclaimed";
                #$memclean = `ipcrm -m $shmem`;
        } else {
                $message = "Memory active";
        }
        print "$shmem - $cpid - $lpid - $cpids - $lpids - $message\n";
}

The output from the script will look similar to the following:

# perl mem_clean.pl
587203562 - 10885 - 17891 - 0 - 0 - Memory active
922746991 - 9728 - 10885 - 0 - 0 - Memory active
150995435 - 9728 - 10885 - 0 - 0 - Memory active
150995432 - 9728 - 17891 - 0 - 0 - Memory active
150995398 - 17421 - 17891 - 1 - 0 - Memory active
150995396 - 13421 - 13891 - 1 - 1 - Memory can be reclaimed
150995387 - 9728 - 10885 - 0 - 0 - Memory active
150995382 - 9728 - 10885 - 0 - 0 - Memory active
150995380 - 9728 - 10885 - 0 - 0 - Memory active
150995379 - 9728 - 10885 - 0 - 0 - Memory active
150995377 - 9728 - 17891 - 0 - 0 - Memory active
150995374 - 9728 - 17891 - 0 - 0 - Memory active
150995371 - 9727 - 10886 - 1 - 0 - Memory active
117440821 - 9728 - 10885 - 0 - 0 - Memory active
117440819 - 9728 - 10885 - 0 - 0 - Memory active

Cleaning Up OpenStack Instances in Redhat Satellite or Spacewalk

When using OpenStack with instances that I wanted to have registered with Redhat Satellite or Spacewalk, I was left wondering what would happen to all those registered hosts once they were terminated in OpenStack?

If I chose to do nothing, the answer was I would be left of orphaned hosts in Redhat Satellite or Spacewalk and over time this could lead to higher license costs if leveraging support for Redhat Linux or just pure database bloat due to having all these previously used instances still referenced in my database.

This issue bothered me and I wanted a mechanism that would cleanup instances one they were terminated but the question was how to go about it?

Well I soon realized that OpenStack keeps a record of all the instances it ever created and or terminated.  It was the terminated part that would be a key component to what I wanted to accomplish.  I figured if I could mine out the deleted data of instances, I could cross check those against Redhat Satellite or Spacewalk.

The Perl script below does just that.   I have it run every 24 hrs out of cron and it first goes into the OpenStack nova database and scrapes the instances table for any instances that were marked deleted in the last 24 hours.   Any instances it finds it puts into an array that I then enumerate through using the spacecmd tools and check within Satellite or Spacewalk to see if the host is registered.  If the host is registered, I then remove the host given that it is no longer a valid host that is up and running.

#!/usr/bin/perl
$cmd=`rm -r -f /root/.spacecmd/spacewalk.schmaustech.com`;
$yestdate=`TZ=CDT+24 /bin/date +%y-%m-%d`;
#$yestdate=`TZ=CDT /bin/date +%Y-%m-%d`;
chomp($yestdate);
@delhosts=`mysql -e "select hostname,uuid,deleted_at from nova.instances where deleted_at is not null"|grep $yestdate`;
foreach $delhost (@delhosts) {
        ($hostname,$uuid) = split(/\s+/,$delhost);
        $uuid2 = $uuid;
        $uuid2 =~ s/-//g;
        @cmdout=`spacecmd -q system_details $hostname.schmaustech.com`;
        foreach $cmd (@cmdout) {
                chomp($cmd);
                if ($cmd =~ /$uuid2/) {
                        $message = "Removing from satellite hostname: $hostname with UUID: $uuid...\n";
                        $cmdtmp = `logger $message`;
                        $cmdtmp = `spacecmd -y -q system_delete $hostname.schmaustech.com`;
                }
        }
}
exit;

Perl and Isilon's REST API: Part 2 Creating Directory

In my last post you saw how I was able to do a directory listing on an Isilon cluster using the Isilon REST API.   In this posting I am providing an example of how you can create a directory on the Isilon cluster using the Isilon REST API.

The simple script below shows how I can create the directory schmaus8 (as an example) under the /ifs/data path on the cluster:

use REST::Client;
use JSON;
use Data::Dumper;
use MIME::Base64;
use IO::Socket::SSL qw( SSL_VERIFY_NONE );
$ENV{PERL_LWP_SSL_VERIFY_HOSTNAME}=0;
my $username = 'admin';
my $password = 'password$';
my $headers = {Accept => 'application/json', Authorization => 'Basic ' . encode_base64($username . ':' . $password), 'x-isi-ifs-target-type' => 'container'};
my $client = REST::Client->new();
$client->getUseragent()->ssl_opts( SSL_verify_mode => 0 );
$client->setHost('https://10.88.82.106:8080');
$client->PUT('/namespace/ifs/data/schmaus8','schmaus8',$headers);
print $client->responseContent(). "\n"; ;
exit;

The output if successful from the script above will be nothing:

C:\TEMP>perl isi-file-create.pl
C:\TEMP>

This script could be altered to provide input for the path you need to have created.  You could also join the previous script I provided to first check and see if the directory exists before you try to create it.

Perl and Isilon's REST API: Part 1 Listing Files In Directory Path

I have been programming in Perl for about 15 years now so anytime I have to write something I usually resort to Perl.   Such was the case when I heard about Isilon and its REST API.   However I soon learned that there is very limited information on Perl and Isilon.

Combing the web I found EMC has a two great manuals that talk about their REST API, the Platform and Namespace manuals (not the official name but provides hints on what to look for).   However these manuals had limited examples and virtually nothing for Perl.   The internet proved to be weak on the subject as well.  I did manage to find some examples with Curl, Python and some read-only type PowerShell scripts.  However I new that whatever I was going to build would at some point need the ability to GET data and PUT/POST data with the REST API.

In the process of my search I did come across a Perl module for Rest::Client.   Seeking out examples of how that module was used with other REST API's and in combination with the EMC manuals for Isilon's REST, I managed to put together my first Perl script.

The following Perl script will list out the files in a directory path on the Isilon cluster:

use REST::Client;
use JSON;
use Data::Dumper;
use MIME::Base64;
use IO::Socket::SSL qw( SSL_VERIFY_NONE );
$ENV{PERL_LWP_SSL_VERIFY_HOSTNAME}=0;
my $username = 'admin';
my $password = 'secret';
my $headers = {Accept => 'application/json', Authorization => 'Basic ' . encode_base64($username . ':' . $password), Content-type => 'application/json'};
my $client = REST::Client->new();
$client->getUseragent()->ssl_opts( SSL_verify_mode => 0 );
$client->setHost('https://10.88.82.106:8080');
$client->GET('/namespace/ifs/data/benz',$headers);
print $client->responseContent(). "\n"; ;
exit;

Here is the output when run from a Windows host:

C:\TEMP>perl isi-file-list.pl
{"children":[{
   "name" : "schmaus3"
}
,{
   "name" : "schmaus1"
}
,{
   "name" : "schmaus2"
}
]}

Basic script that we could then add argument parameters to to fill in the host, username, password and path details.   We could then also write something to parse the JSON response and make it look nice and neat.   Future version, but the core of the example shows you how to do the request in Perl.

Next post will be how you can create a directory on the Isilon cluster.

Using Netapp DFM and Perl to Manage Netapp Filer /etc Files

Sometimes you have to manage the /etc/passwd and /etc/group files on your Netapp filer and seemingly the only options available are to use rdfile and wrfile or a text editor like vi via a NFS mount or Notepad++ via a CIFS share.   None of which appeal to me when trying to create something that a less technical person could use to manipulate these files.

Below is the rough framework that could be used to build a full fledged file manipulator for Netapp files under the /etc directory.  In the example below we are looking at the /etc/passwd file, however it could be expanded to manipulate any files on the filer through DFM.   Further, you could use Win32:GUI or Perl/TK to provide a GUI to the script as opposed to running it via the command line.

The breakdown of the script is as follows:

Standard Perl interpreter line.  In this example we are on Windows.

     #!/Perl64/bin/perl

This section of the script is a basic variable assignment of what I want my new line to be in the /etc/passwd file.  However you could have an input prompt here and/or have it read from a file.

     $newentry = "Your passwd entry\n";

This line grabs the existing /etc/passwd file and loads it into a perl array called rpasswd.  It is using the DFM command set to run rdfile on the filer.

     chomp (@rpasswd = `dfm run cmd -t 120 faseslab1 rdfile /etc/passwd`);

  This section cleans up the rpasswd values and only puts in the lines that match Stdout from the DFM output into a second array called passwd.

    foreach $line (@rpasswd) {

                     if ($line =~ /^Stdout:\s+/) {

                                     $line =~ s/^Stdout:\s+//g;

                                     push(@passwd,$line);

                     }

     }

This line  places the new entry into the passwd array.

     push (@passwd,$newentry);

This line backs up the existing passwd file using DFM and mv command.

     $result = `dfm run cmd -t 120 faseslab1 mv /etc/passwd /etc/passwd.bak`;

This line writes the new passwd file using the passwd array and DFM to write out the new file.

     foreach $line (@passwd) {

                     $result = `dfm run cmd -t 120 faseslab1 wrfile -a /etc/passwd $line`;

     }

     exit;

 Again, this is a rough example, but it gives you the idea of what can be done using Perl and DFM.

Configuring Persistent Bindings on Solaris 10

If you have tape devices attached to your Solaris 10 host and you often find that after a reboot of the host, the tape devices are no longer in the same order they were before, you can use the following Perl script to configure the /etc/devlink.tab file to make the tape devices persist.  Script is below:

#!/usr/bin/perl
#################################################################
# This script maps fiber attached tape drives to persistently   #
# bind to the same device across reboots.                       #
# (C) 2011 Benjamin Schmaus                                     #
#################################################################
use strict;
my($junk,$path,$devices,$dev,$file);
my(@devices,@file);
my $date = `date +%m%d%Y`;
$file = `/usr/bin/cp /etc/devlink.tab /etc/devlink.tab.$date`;
@file = `cat /etc/devlink.tab`;
@file = grep !/type=ddi_byte:tape/, @file;
open (FILE,">/etc/devlink.tab.new");
print FILE @file;
close (FILE);
@devices = `ls -l /dev/rmt/*cbn|awk {'print \$9 \$11'}`;
open (FILE,">>/etc/devlink.tab.new");
foreach $devices (@devices) {
                chomp($devices);
                ($dev,$path) = split(/\.\.\/\.\./,$devices);
                $dev =~ s/cbn//g;
                $dev =~ s/\/dev\/rmt\///g;
                $path =~ s/:cbn//g;
                ($junk,$path) = split(/st\@/,$path);
                print FILE "type=ddi_byte:tape;addr=$path;\trmt/$dev\\M0\n";
}
close (FILE);
$file = `/usr/bin/mv /etc/devlink.tab.new /etc/devlink.tab`;      
exit

Expire List of Tapes in Netbackup

When working in a large Netbackup environment, there often comes a time when you need to expire a large amount of tapes all at once.

This was exactly the scenario that happened to me when a company I consulted at changed their retention periods.  The change of retention periods from 1 year to 1 month meant they wanted to free up all the tapes that contained images that were more then a month old.

The solution to the problem was a script that basically does the following:

1) Reads in a list of media id's from a file.
2) Determine which media server the media id is assigned.
3) Expire the media id from the Netbackup catalog.

The script is here:

#!/usr/bin/perl
$master="hostname of master server";
open DATA, "cat /home/schmaubj/media.list|";
while () {
 $mediaid = $_;
 chomp($mediaid);
 open DATA2, "/usr/openv/netbackup/bin/admincmd/bpmedialist -U -m $mediaid|";
 while () {
  $line = $_;
  chomp($line);
  if ($line =~ /Server Host/) {
   ($junk,$mhost) = split(/=/,$line); 
   chomp($mhost);
    $mhost =~ s/ *$//;  
          $mhost =~ s/^ *//; 
  }
 }
 close (DATA2);
 print "Media ID: $mediaid\n";
 print "Media Server Host: $mhost\n";
 print "Expiring now...\n";
 $expire=`/usr/openv/netbackup/bin/admincmd/bpexpdate -force -d 0 -m $mediaid -host $mhost -M $master`;
}
close (DATA);

Netbackup Auto Discovery of VM's

When I first started to work with Netbackup and VMWare VCB backups, the biggest complaint I often found was that Netbackup could not auto discover new VM's and automatically insert them into a given policy. Along with that I often found that if I was running more then one VCB backup against my Netbackup proxy server, and one or more of those VM's resided on the same VMWare datastore, I would often get a snapshot error, as both tried to compete to get a snapshot on the datastore. These issues made Netbackup VCB backups problematic and management intensive for the large environment of 50+ VM's.

Pondering this issue, I came up with an elegant yet useful solution. The idea was to create multiple policies, one for each datastore in my VMWare cluster. The policy would be configured to run one client at a time, and all the policies would run at the same time. In the end you would end up with VCB backup jobs running against the proxy server, but they would not be on the datastore since each policy was datastore specific. This eliminated the snapshot errors and also allowed for me to have the policies auto updated.

Like I said, I was able to have my policies auto updated with new VM clients without any intervention. This was made possible by writing a script that used the VMWare Perl API's and made calls into both VMWare and then Netbackup. Here is how the setup works:

1) Create a Netbackup policy for each datastore in your VMWare environment. Make sure the policies all have a naming convention. Example: NB_VCB_datastore1, NB_VCB_datastore2, NB_VCB_datastore3, etc...
2) Use the script below and run from the Unix/Linux master server or any Unix/Linux server that has the Netbackup admincmd directories installed and is allowed access to the Netbackup master server. Make sure you have the VMWare Perl API libraries installed with Perl on the master server.

Usage: autovcb -v (vc server) -u (username) -p (password) -d (datacenter in vc) -os (Windows|Solaris|Linux|Suse|Redhat) -pp (netbackup policy prefix) -m (master server) -dsp (datastore name|ALL) [-pre (datastore match prefix)] [-update]

Note: The Netbackup policy prefix is the naming convention value you used before the datastore in the policy name. In my example it is "NB_VCB_". Also, be aware that forward and reverse DNS must be functional on the VM hosts to ensure proper inclusion in the policy.

The script allows you to only match on some datastore or all datastores. You can do a full discovery and populate or just an update of the policies. You can also chose to only add clients of a specific OS.

This script was tested in a Netbackup 6.5.4 and VMWare ESX 3.5 environment.

The script is here:

#!/usr/bin/perl
#################################################################################
# Script that autodiscovers VM's and places the client name into prexisting #
# policies based on the datastore names discovered    #
# (C) Benjamin Schmaus May, 2009      #
################################################################################# 
use strict;
use warnings;
use Socket;
use VMware::VIRuntime;
use VMware::VILib;
use VMware::VIM25Stub;
use Getopt::Long;

### Main Logic ###
my $bpplclients = '/usr/openv/netbackup/bin/admincmd/bpplclients';
my ($help,$prefix,$vcserver,$username,$password,$dc,$os,$polpr,$datastorepol,$update,$master);
my ($datastorepoltmp,$listds,$esxhost,$vmname,$adhostname,$ipaddress,$toolstat,$guestos,$datastore,$hostname,$junk,$junk2,$junk3,$client,$datastorefix,$polname,$gueststate);
my @dslist = ();
options();
print "update is set to: $update\n";
my $policy = "$polpr$datastorepol";
my $url = "https://$vcserver/sdk/vimService";
Vim::login(service_url => $url, user_name => $username, password => $password);
my $datacenter_views = Vim::find_entity_views(view_type => 'Datacenter',filter => { name => $dc });
my $dcds = Vim::find_entity_view(view_type => "Datacenter", filter => {'name' => "Fridley"} );
my $ds = $dcds->datastore;

if ($datastorepol eq "ALL") {
 getlistds();
 print "\n";
}
if ($update eq "0") {
 polrmclients();
 print "\n";
}

if ($datastorepol eq "ALL") {
 print "Discovering new clients...\n";
 # go through all policies and add new clients for all datastores
 foreach (@dslist) {
  $datastorepoltmp = $_; 
  print "\tDatastore: $datastorepoltmp\n";
  print "\tPolicy: $polpr$datastorepoltmp\n";
  polgetclients();
  print "\t\n";
 }
} else {
 print "Discovering new clients...\n";
 # go through all policies and add new clients for given datastore
 $datastorepoltmp = $datastorepol;
 print "\tCurrent Datastore: $datastorepoltmp\n";
 print "\tCurrent Policy: $$polpr$datastorepoltmp\n";
 polgetclients();
 print "\t\n";
}
print "\n\n";
Vim::logout();
exit;


sub polgetclients {
 foreach (@$datacenter_views) {
  my $datacenter = $_->name;
  my $host_views = Vim::find_entity_views(view_type => 'HostSystem',begin_entity => $_ );
  foreach (@$host_views) {
   $esxhost = $_->name;
   #print "\tESX Host Server: $esxhost\n";
   my $vm_view = Vim::find_entity_views(view_type => 'VirtualMachine',begin_entity => $_ , filter => { 'guest.guestState' => 'running' });
   foreach (@$vm_view) {
    $vmname = $_->name;
    $adhostname = $_->summary->guest->hostName;
    $ipaddress = $_->summary->guest->ipAddress;
    $toolstat = $_->summary->guest->toolsStatus->val;
    $guestos = $_->summary->guest->guestFullName;
    $datastore = $_->summary->config->vmPathName;
    $gueststate = $_->guest->guestState;
    ($datastorefix,$junk) = split(/\] /,$datastore);
    $datastorefix =~ s/\[//g;
    $datastorefix =~ s/ /_/g;
    if ($ipaddress) {
     $hostname = gethostbyaddr(inet_aton($ipaddress), AF_INET);
     #print "$ipaddress $hostname\n";
     #if (!$hostname) { $hostname = '---'; }
     #$hostname =~ s/.asd.udlp.com/.mpls.udlp.com/g;
     #if (($hostname !~ /.mpls.udlp.com/) && ($hostname !~ /udlp.com/)) {
     # $hostname = "$hostname.mpls.udlp.com";
     #}
    }
    if (!$hostname) { $hostname = '---'; }
    $policy = "$polpr$datastorefix";
    if ($datastorefix eq $datastorepoltmp) {
     if ($toolstat =~ /toolsOk/) {
      if ($guestos =~ /$os/) {
       if ($hostname ne "---" && $hostname ne "localhost") {
        my $addtopolicy = `$bpplclients $policy -M $master -add $hostname VMware Virtual_Machine > /dev/null 2>&1`;
        if ($? eq "0") {
         print "\t\tAdded $hostname\n";
        } else {
         print "\t\tSkipped $hostname: Already in policy\n";
        }
       } else {
        print "\t\tSkipped $vmname: Reverse DNS does not exist\n";
       }
      } else {
       print "\t\tSkipped $vmname: OS does not match guestos\n";
      }
     } else {
      print "\t\tSkipped $vmname: Tools not okay - $toolstat\n";
     }
    }
   }
  }
 }
}

sub polrmclients {
 if ($datastorepol eq "ALL") {
  foreach (@dslist) {
   $policy = "$polpr$_";
   print "Removing clients from $policy...\n";
   my @polrm = `$bpplclients $policy -noheader`;
   foreach (@polrm) {
    ($junk,$junk2,$client) = split(/ /,$_);
    chomp($client);
    my $remove = `$bpplclients $policy -M $master -delete $client > /dev/null 2>&1`;
    if ($? eq "0") {
                                 print "\t\tRemoved $client\n";
                                } else {
                                 print "\t\tSkipped $client: Not in policy\n";
    }
   }
  }
 } else { 
  $policy = "$polpr$datastorepol";
  my @polrm = `$bpplclients $policy -noheader`;
  print "Removing clients from $policy...\n";
  foreach (@polrm) {
   ($junk,$junk2,$client) = split(/ /,$_);
   chomp($client);
   my $remove = `$bpplclients $policy -M $master -delete $client > /dev/null 2>&1`;
   if ($? eq "0") {
                         print "\t\tRemoved $client\n";
                        } else {
                         print "\t\tSkipped $client: Not in policy\n";
                        }
  }
 }
}

sub getlistds {
 print "Getting list of datastores...\n";
 my $counter = 0;
 foreach(@$ds) {
         my $ds_ref = Vim::get_view(mo_ref => $_);
         my $ds = $ds_ref->info->name;
         if (($ds =~ /$prefix/) && ($prefix ne "---")) {
                 $dslist[$counter] = $ds_ref->info->name;
                 $dslist[$counter] =~ s/ /_/g;
   $dslist[$counter] =~ s/\(|\)//g;
   print "\tFound datastore: $dslist[$counter]\n";
                 ++$counter;
         } elsif  ($prefix eq "---") {
                 $dslist[$counter] = $ds_ref->info->name;
                 $dslist[$counter] =~ s/ /_/g;
   $dslist[$counter] =~ s/\(|\)//g;
   print "\tFound datastore: $dslist[$counter]\n";
                 ++$counter;
         }
 }
}

sub options {
        $vcserver="";$username="";$password="";$dc="";$os="";$polpr="";$master="";$update="0";$prefix="";$datastorepol="";
        GetOptions ('h|help'=>\$help,'v|vcserver=s'=>\$vcserver,'u|username=s'=>\$username,'p|password=s'=>\$password,'d|datacenter=s'=>\$dc,'os=s'=>\$os,'pp|polpr=s'=>\$polpr,'m|master=s'=>\$master,'dsp=s'=>\$datastorepol,'pre|prefix=s'=>\$prefix,'update'=>\$update);
        if ($help) {
                print "Usage: autovcb -v  -u  -p  -d  -os  -pp  -m  -dsp <(datastore name|ALL)> [-pre ] [-update]\n";
                exit;
        }
        if (($vcserver eq "") || ($username eq "") || ($password eq "") || ($dc eq "") || ($os eq "") || ($polpr eq "") || ($master eq "") || ($datastorepol eq "")) {
  print "Missing required parameters - Type -help for options\n";
                exit;
        }
 if (($prefix) && ($datastorepol ne "ALL")) {
  print "Cannot use prefix option when datastore is not set to ALL - Type -help for options\n";
  exit;
 }

        if (($os ne "Windows") && ($os ne "Solaris") && ($os ne "Linux") && ($os ne "Suse") && ($os ne "Redhat")) {
  print "Incorrect OS specified - Type -help for options\n";
                exit;
        }
}

VMWare Tool Check

If you have ever worked in a large VMWare environment, you know what a pain it can be to determine if the VMWare tools are up to date on a VMWare host. That is why I wrote the following Perl script. The script enumerates through the VMWare hosts and tells you if the VMWare tools are up to date or not.

#!/usr/bin/perl
#################################################################################
# Script that prints out status of VMware tools for given DataCenter		#
# (C) Benjamin Schmaus July, 2009						#
################################################################################# 
use strict;
use warnings;
use Socket;
use VMware::VIRuntime;
use VMware::VILib;
use VMware::VIM25Stub;
use Getopt::Long;

### Main Logic ###
my ($help,$prefix,$vcserver,$username,$password,$dc);
my ($esxhost,$vmname,$adhostname,$ipaddress,$toolstat,$guestos,$datastore,$gueststate);
options();
my $url = "https://$vcserver/sdk/vimService";
Vim::login(service_url => $url, user_name => $username, password => $password);
my $datacenter_views = Vim::find_entity_views(view_type => 'Datacenter',filter => { name => $dc });
my $dcds = Vim::find_entity_view(view_type => "Datacenter", filter => {'name' => $dc } );
my $ds = $dcds->datastore;
getclients();
print "\n\n";
Vim::logout();
exit;

sub getclients {
	foreach (@$datacenter_views) {
		my $datacenter = $_->name;
		my $host_views = Vim::find_entity_views(view_type => 'HostSystem',begin_entity => $_ );
		foreach (@$host_views) {
			$esxhost = $_->name;
			my $vm_view = Vim::find_entity_views(view_type => 'VirtualMachine',begin_entity => $_ , filter => { 'guest.guestState' => 'running' });
			foreach (@$vm_view) {
				$vmname = $_->name;
				#print "$vmname\n";
				$adhostname = $_->summary->guest->hostName;
				$ipaddress = $_->summary->guest->ipAddress;
				$toolstat = $_->summary->guest->toolsStatus->val;
				$guestos = $_->summary->guest->guestFullName;
				$datastore = $_->summary->config->vmPathName;
				$gueststate = $_->guest->guestState;
				if ($toolstat !~ /toolsOk/) {
					print "TOOLS STATUS: $toolstat\tVM: $vmname\n";
				}
			}
		}
	}
}

sub options {
        $vcserver="";$username="";$password="";$dc="";
        GetOptions ('h|help'=>\$help,'v|vcserver=s'=>\$vcserver,'u|username=s'=>\$username,'p|password=s'=>\$password,'d|datacenter=s'=>\$dc);
        if ($help) {
                print "Usage: vmtoolchk.pl -v  -u  -p  -d  \n";
                exit;
        }
        if (($vcserver eq "") || ($username eq "") || ($password eq "") || ($dc eq "")) {
		print "Missing required parameters - Type -help for options\n";
                exit;
        }
}

Perl, VMWare Snapshots & Netbackup

In VmWare, Virtual Center has the ability for you to take snapshots. These might come in handy before you install a new application, perform a OS upgrade or make any other significant change to the VM.

Netbackup also creates a snapshot when you do a VCB backup or incremental backup against a VM. Sometimes these snapshots are left behind when a backup of the VM fails. The result is that you have to go into Virtual Center, find the VM and remove the snapshot. This can be time consuming and lead me to develop a tool in Perl as a solution.

The Perl solution I came up with was inspired by the sample snapshot Perl script that comes with the VMWare Perl API SDK. This script however was not specific for my needs. So I came up with a script that would delete snapshots based on a pattern match of the name of the snapshot.

Basically the script will connect to a Virtual Center server and enumerate through the hosts associated with that Virtual Center. While enumerating through the hosts, it checks for snapshots and then compares the pattern entered to the name of the snapshot. If they match it will remove it. In the case of a Netbackup VCB backup snapshot, the names always start with "VCB_". Thus if you use "VCB_" as your pattern you can remove all snapshots on all your hosts. You can extend this to all snapshots if you use a naming convention when you create snaps.

The script is here:

#!/usr/bin/perl
#################################################################################
# Script that autodiscovers VM's and places the client name into prexisting     #
# policies based on the datastore names discovered                              #
# (C) Benjamin Schmaus May, 2009                                                #
#################################################################################
use strict;
use warnings;
use Socket;
use VMware::VIRuntime;
use VMware::VILib;
use VMware::VIM25Stub;
use Getopt::Long;
my ($help,$vcserver,$username,$password,$dc,$removesnapshot,$children,$snapshotname);
$children="0";
options();
my $url = "https://$vcserver/sdk/vimService";
Vim::login(service_url => $url, user_name => $username, password => $password);
my $datacenter_views = Vim::find_entity_views(view_type => 'Datacenter',filter => { name => $dc });
list_snapshot();
exit;

sub list_snapshot {
 my $datacenter_views = Vim::find_entity_views(view_type => 'Datacenter',filter => { name => $dc });
    foreach (@$datacenter_views) {
  my $datacenter = $_->name;
  my $host_views = Vim::find_entity_views(view_type => 'HostSystem',begin_entity => $_ );
  foreach (@$host_views) {
   my $esxhost = $_->name;
   my $vm_view = Vim::find_entity_views(view_type => 'VirtualMachine',begin_entity => $_ , filter => { 'guest.guestState' => 'running' });
   foreach (@$vm_view) {
          my $mor_host = $_->runtime->host;
          my $hostname = Vim::get_view(mo_ref => $mor_host)->name;
    my $ref = undef;
    my $nRefs = 0;
          my $count = 0;
          my $snapshots = $_->snapshot;
          if(defined $snapshots) {
             Util::trace(0,"\nSnapshots for Virtual Machine ".$_->name . " under host $hostname\n");
             printf "\n%-47s%-16s %s %s\n", "Name", "Date","State", "Quiesced";
             print_tree ($_->snapshot->currentSnapshot, " ", $_->snapshot->rootSnapshotList);
     if(defined $_->snapshot) {
                                         ($ref, $nRefs) = find_snapshot_name ($_->snapshot->rootSnapshotList, $snapshotname);
                                 }
     if ($snapshotname =~ /$removesnapshot/) {
      print "Snapshot name: $snapshotname matches for delete\n";

            if (defined $ref && $nRefs == 1) {
               my $snapshot = Vim::get_view (mo_ref =>$ref->snapshot);
               eval {
                   $snapshot->RemoveSnapshot (removeChildren => $children);
                    Util::trace(0, "\nRemove Snapshot ". $snapshotname . " For Virtual Machine ". $_->name . " under host $hostname" ." completed sucessfully\n");
       };
               if ($@) {
                   if (ref($@) eq 'SoapFault') {
                       if(ref($@->detail) eq 'InvalidState') {
                          Util::trace(0,"\nOperation cannot be performed in the current state of the virtual machine");
                       } elsif(ref($@->detail) eq 'HostNotConnected') {
                          Util::trace(0,"\nhost not connected.");
                       } else {
                          Util::trace(0, "\nFault: " . $@ . "\n\n");
                       }
                   } else {
                       Util::trace(0, "\nFault: " . $@ . "\n\n");
                   }
               }
            } else {
               if ($nRefs > 1) {
                   Util::trace(0,"\nMore than one snapshot exits with name" ." $snapshotname in Virtual Machine ". $_->name ." under host ". $hostname ."\n");
               }
               if($nRefs == 0 ) {
                   Util::trace(0,"\nSnapshot Not Found with name" ." $snapshotname in Virtual Machine ". $_->name ." under host ". $hostname ."\n");
               }
      }
     }
          #} else {
            # Util::trace(0,"\nNo Snapshot of Virtual Machine ".$_->name ." exists under host $hostname\n");
          }
   }
  }
 }
}

sub options {
        $vcserver="";$username="";$password="";$dc="";$removesnapshot="";
        GetOptions ('h|help'=>\$help,'v|vcserver=s'=>\$vcserver,'u|username=s'=>\$username,'p|password=s'=>\$password,,'d|datacenter=s'=>\$dc,'sm|snapmatch=s'=>\$removesnapshot);
        if ($help) {
                print "Usage: snapper.pl -v  -u  -p  -d  -sm \n";
                exit;
        }
        if (($vcserver eq "") || ($username eq "") || ($password eq "") || ($dc eq "") || ($removesnapshot eq "")) {
                print "Missing required parameters - Type -help for options\n";
                exit;
        }

}

sub print_tree {
 my ($ref, $str, $tree) = @_;
    my $head = " ";
    foreach my $node (@$tree) {
        $head = ($ref->value eq $node->snapshot->value) ? " " : " " if (defined $ref);
        my $quiesced = ($node->quiesced) ? "Y" : "N";
        $snapshotname = $node->name;
        printf "%s%-48.48s%16.16s %s %s\n", $head, $str.$node->name,
              $node->createTime, $node->state->val, $quiesced;
        print_tree ($ref, $str . " ", $node->childSnapshotList);
    }
    return;
}

sub find_snapshot_name {
 my ($tree, $name) = @_;
    my $ref = undef;
    my $count = 0;
    foreach my $node (@$tree) {
        if ($node->name eq $name) {
           $ref = $node;
           $count++;
        }
        my ($subRef, $subCount) = find_snapshot_name($node->childSnapshotList, $name);
        $count = $count + $subCount;
        $ref = $subRef if ($subCount);
    }
    return ($ref, $count);
}

Netapp Aggregate/Volume Report

Getting information from the Netapp is easy with the web interface and the command line interface. However sometimes we may want information in a Excel format.  Further we may want it automatically generated and emailed to certain recipients.

This was the case when I created the following Perl script that gathers all the volume/aggregate usage information from a Netapp filer or group of Netapp filers and sends that information via SMTP in a easy to read Excel formatted report.

The script is below and shows that it will run against 4 different Netapp filers that all have the same password and then email to bschmaus@domain.com.   Note that all those variables are examples and need to be updated when using in a real environment.

#!/usr/bin/perl
#########################################################
# Netapp Aggregate Report in Excel Format               #
#########################################################
use strict;
use Net::SSH::Perl;
use Spreadsheet::WriteExcel;
### Variables ###
my @hosts = ("netapp1","netapp2","netapp3","netapp4");
my $user="root";my $password="pasword";
my $subject = "Netapp_Space_Utilization_Report";
my @emails = ('bschmaus\@domain.com');
my (%saw,$scon,$exit,$errors,$count,$output,$hosts,$tmparray,$aggr,$state,$value);
my ($output2,$host,$flag,$vola,$volu,$volav,$aggrts,$aggrwr,$aggrsr,$aggrus,$aggrbn,$vola,$volu,$volg,$junk1,$junk2);
my ($mailit,$volpu,$message,$emails,$format);
my $sc1="2";my $sc2="2";my $sc3="2";
my (@output,@tmparray,$tmpaggrs,@tmpaggrs,@aggrs);
### Program Logic ### 

### Setup Excel Worksheet ###
my $workbook  = Spreadsheet::WriteExcel->new('/tmp/netappvsr.xls');
$workbook->compatibility_mode();
my $worksheet = $workbook->add_worksheet('Aggregates');
my $worksheet = $workbook->add_worksheet('Volumes');
my $worksheet = $workbook->add_worksheet('Volume Usage in Aggregate');
my $format = $workbook->add_format();
my $header = $workbook->add_format();
create_excel();

foreach $hosts (@hosts) {
        print "Gathering aggregates from $hosts...\n";
        $scon = Net::SSH::Perl->new ("$hosts",protocol => 2);
        $scon->login("$user","$password");
        ($output[$count],$errors,$exit) = $scon->cmd("aggr status;logout telnet");
	#print "$output[$count]\n";
        $count++;
}
$count = 0;
foreach $output (@output) {
        @tmparray = split(/\n/,$output);
	#shift(@tmparray);
	#pop(@tmparray);
        foreach $tmparray (@tmparray) {
		if ($tmparray =~ /aggr/) {
               		($aggr,$state) = split(' ',$tmparray);
			$value="$hosts[$count]:$aggr:$state";
                	push(@tmpaggrs,$value);
		}
        }
	$count++;
}

print "Sorting aggregates from Netapps...\n";
undef %saw;
@saw{@tmpaggrs} = ();
@aggrs = sort keys %saw;
foreach (@aggrs) {
	($host,$aggr,$state) =  split(/:/);
	#print "HOST: $host AGGR: $aggr STATE: $state\n";
	print "Gathering $aggr detail on $host...\n";
        $scon = Net::SSH::Perl->new ("$host",protocol => 2);
        $scon->login("$user","$password");
        ($output2,$errors,$exit) = $scon->cmd("aggr show_space -g $aggr;logout telnet");
	@tmparray = split(/\n/,$output2);
	foreach $tmparray (@tmparray) {
		if (($tmparray =~ /Aggregate/) && ($tmparray =~ /\'/)) { 
			$flag = "1";
		}
		if (($tmparray =~ /GB/) && ( $flag eq "1" )) {
			($aggrts,$aggrwr,$aggrsr,$aggrus,$aggrbn) = split(' ',$tmparray);
			#print "$aggrts,$aggrwr,$aggrsr,$aggrus,$aggrbn\n";
			$workbook->sheets(0)->write($sc1,0,$host);
			$workbook->sheets(0)->write($sc1,1,$aggr);
			$workbook->sheets(0)->write($sc1,2,$aggrts);
			$workbook->sheets(0)->write($sc1,3,$aggrwr);
			$workbook->sheets(0)->write($sc1,4,$aggrsr);
			$workbook->sheets(0)->write($sc1,5,$aggrus);
			$sc1++;
			
		}
		if (($tmparray =~ /Volume/) && ($tmparray =~ /Allocated/)) {
			$flag="2";
		}
		if (($tmparray =~ /vol/) && ($tmparray =~ /GB/) && ( $flag eq "2" )) {
			($vola,$volu,$volg) = split(' ',$tmparray);
			#print "$vola,$volu,$volg\n";
			$workbook->sheets(1)->write($sc2,0,$host);
			$workbook->sheets(1)->write($sc2,1,$vola);
			$workbook->sheets(1)->write($sc2,2,$volu);
			$workbook->sheets(1)->write($sc2,3,$volg);
			if ($volu eq "0GB") {
				$volpu = "0";
			} else {
				$volpu = int(($volg/$volu)*100) + 1;
			}
			$workbook->sheets(1)->write($sc2,4,$volpu);
			$sc2++;
			

		}
		if (($tmparray =~ /Aggregate/) && ($tmparray =~ /Allocated/)) {
			$flag="3"; 
		}
		if ((($tmparray =~ /Total space/) || ($tmparray =~ /Snap reserve/) || ($tmparray =~ /WAFL/)) && ($flag eq "3")) {
			($junk1,$junk2,$vola,$volu,$volav) = split(/\s+/,$tmparray);
			#print "$junk1,$junk2,$vola,$volu,$volav\n";
			$workbook->sheets(2)->write($sc3,0,$host);
                        $workbook->sheets(2)->write($sc3,1,$aggr);
			if ($vola eq "0GB") {
				$volpu = "0";
			} else {
				$volpu = int(($volu/$vola)*100) + 1;
			}
			if ($tmparray =~ /Total space/) {
				$workbook->sheets(2)->write($sc3,2,$vola);
				$workbook->sheets(2)->write($sc3,3,$volu);
				$workbook->sheets(2)->write($sc3,4,$volpu);
			}
			if ($tmparray =~ /Snap reserve/) {
                                $workbook->sheets(2)->write($sc3,5,$vola);
                                $workbook->sheets(2)->write($sc3,6,$volu);
                                $workbook->sheets(2)->write($sc3,7,$volpu);
			}
			if ($tmparray =~ /WAFL/) {
                                $workbook->sheets(2)->write($sc3,8,$vola);
                                $workbook->sheets(2)->write($sc3,9,$volu);
                                $workbook->sheets(2)->write($sc3,10,$volpu);
				$sc3++;
			}
				 
		}
	}
}	

$workbook->close();
$workbook->close();

### Mail Off Results ###
mailit();
exit;

### Setup Excel Format Subroutine ###
sub create_excel {
        $format->set_bold();
        $format->set_size(16);
        $format->set_align('center');
        $header->set_bold();
        $header->set_align('center');
        $workbook->sheets(0)->set_column(0, 5, 20);
	$workbook->sheets(0)->write(1, 0,  'Host', $header);
        $workbook->sheets(0)->write(1, 1,  'Aggregate', $header);
        $workbook->sheets(0)->write(1, 2,  'Total Space(GB)', $header);
        $workbook->sheets(0)->write(1, 3,  'WAFL Reserve(GB)', $header);
        $workbook->sheets(0)->write(1, 4,  'Snap Reserve(GB)', $header);
        $workbook->sheets(0)->write(1, 5,  'Usable Space(GB)', $header);
        $workbook->sheets(1)->set_column(0, 4, 20);
	$workbook->sheets(1)->write(1, 0,  'Host', $header);
        $workbook->sheets(1)->write(1, 1,  'Volume', $header);
        $workbook->sheets(1)->write(1, 2,  'Allocated(GB)', $header);
        $workbook->sheets(1)->write(1, 3,  'Used(GB)', $header);
        $workbook->sheets(1)->write(1, 4,  '%Used', $header);
        $workbook->sheets(2)->set_column(0, 10, 25);
	$workbook->sheets(2)->write(1, 0,  'Host', $header);
        $workbook->sheets(2)->write(1, 1,  'Aggregate', $header);
        $workbook->sheets(2)->write(1, 2,  'Total Space Allocated(GB)', $header);
        $workbook->sheets(2)->write(1, 3,  'Total Space Used(GB)', $header);
        $workbook->sheets(2)->write(1, 4,  '%Total Space Used', $header);
        $workbook->sheets(2)->write(1, 5,  'Snap Reserve Allocated(GB)', $header);
        $workbook->sheets(2)->write(1, 6,  'Snap Reserve Used(GB)', $header);
        $workbook->sheets(2)->write(1, 7,  '%Snap Reserve Used', $header);
        $workbook->sheets(2)->write(1, 8,  'WAFL Reserved Allocated(GB)', $header);
        $workbook->sheets(2)->write(1, 9,  'WAFL Reserved Used(GB)', $header);
        $workbook->sheets(2)->write(1, 10,  '%WAFL Reserved Used', $header);
        $workbook->sheets(0)->merge_range('A1:F1','NetApp Storage Report',$format);
        $workbook->sheets(1)->merge_range('A1:E1','NetApp Storage Report',$format);
        $workbook->sheets(2)->merge_range('A1:I1','NetApp Storage Report',$format);
}

### Mail Subroutine ###
sub mailit {
        $message = `echo "NetApp Storage Report Attached">/tmp/nvsr-body.txt`;
        $message = `echo "">>/tmp/nvsr-body.txt`;
        $message = `/usr/bin/uuencode /tmp/netappvsr.xls netappvsr.xls > /tmp/nvsr-attachment.txt`;
        $message = `cat /tmp/nvsr-body.txt /tmp/nvsr-attachment.txt > /tmp/nvsr.txt`;
        foreach $emails (@emails) {
                $mailit = `/usr/bin/mailx -s $subject $emails < /tmp/nvsr.txt`;
        }
}

Netapp Coverage Report for Netbackup

Anyone who has used Netbackup should be familiar with the coverage report script provided in the goodies directory. The script provides the ability to see what filesystems are being backed up on the clients in your Netbackup policies. The problem with the script is that it relies on accessing clients that have the Netbackup client installed on them. This works great except when it comes to NDMP backup policies for Netapps.

The reality is that the coverage report supplied does not provide details on what volume paths are being covered and which ones are being missed when using Netbackup to backup a Netapp filer. However, after being approached to write a solution to the issue, I came up with a script that provides that missing coverage information.

The script I wrote is Perl based and at this point needs to be run from a Unix/Linux master server (may work on Windows but was never tested). The script, when edited and provided with the proper parameters, will go out and gather the path information from one or more Netapp filers and then compare those hosts and paths to what you have in your Netbackup polices for NDMP backups. The results are then emailed off in an Excel formatted spreadsheet.

The script is here:

#!/usr/bin/perl
#########################################################
# Netapp Netbackup Coverage Report                      #
#########################################################
use strict;
use Net::SSH::Perl;
use Spreadsheet::WriteExcel;
my @hosts = ("netapp1","netapp2");
my (@output);my $hcount = 0;my $user="root";my $password="password";
my (@netpaths,@tmpnetpaths,$host,$tmparray,$output,$name,$path,$comment,$coverage);
my $subject = "Netapp_Backup_Coverage_Report";
my @emails = ('benjamin.schmaus\@schmaustech.com');
my (%saw,$scon,$errors,$exit,$netpaths,@tmparray);
my ($include,$policyname,$junk,@ppaths,$ppaths,$loop);
my ($emails,$mailit,$message);
my $num="2";

### Setup Excel Worksheet ###
my $workbook  = Spreadsheet::WriteExcel->new('/tmp/ncoverage.xls');
my $worksheet = $workbook->add_worksheet();
my $format = $workbook->add_format();
my $header = $workbook->add_format();
create_excel();

foreach $host (@hosts) {
 print "Gathering paths from $host...\n";
 $scon = Net::SSH::Perl->new ("$host",protocol => 2);
 $scon->login("$user","$password");
 ($output[$hcount],$errors,$exit) = $scon->cmd("cifs shares;logout telnet");
 $hcount++;
}
foreach $output (@output) {
 @tmparray = split(/\n/,$output);
 foreach $tmparray (@tmparray) {
  if ($tmparray =~ /\/vol/) {
   ($name,$path,$comment) = split(' ',$tmparray);
   $path =~ tr/[A-Z]/[a-z]/;
   push(@tmpnetpaths,$path);
  }
 }
}
print "Sorting paths from Netapps...\n";
undef %saw;
@saw{@tmpnetpaths} = ();
@netpaths = sort keys %saw;
foreach $host (@hosts) {
 print "Gathering backup selections from Netbackup for $host...\n";
        open DATA, "/usr/openv/netbackup/bin/admincmd/bppllist -allpolicies -U -byclient $host|";
        while () {
                chomp();
  $_ =~ s/\s//g;
  if ($_ =~ /PolicyName/) {
                        ($junk,$policyname) = split(/:/);
  }
  if ($_ =~ /\/vol\//) {
                 $_ =~ s/\s//g;
   $_ =~ s/Include://g;
   $_ =~ s/\/*$//g;
   $_ =~ tr/[A-Z]/[a-z]/;
   push (@ppaths,"$policyname:$_");
  }
        }
        close (DATA);
}
#### Reconcile time ####
print "Reconcile paths...\n";
foreach $netpaths (@netpaths) {
 $path = $netpaths;
 $loop ="0";
 foreach $ppaths (@ppaths) { 
  ($policyname,$include) = split(/:/,$ppaths);
  if ($netpaths =~ /$include/) {
   $coverage="COVERED";
   cell();
   $loop ="1";
  }
 }
 if ($loop eq "0") {
  $coverage="UNCOVERED";
  $policyname="NONE";
  cell();
 }
}

$workbook->close();


### Mail Off Results ###
print "Mailing off results.\n";
mailit();
exit;

### Setup Excel Format Subroutine ###
sub create_excel {
        $format->set_bold();
        $format->set_size(16);
        $format->set_align('center');
        $header->set_bold();
        $header->set_align('center');
        $worksheet->set_column(0, 0, 40);
        $worksheet->set_column(1, 1, 20);
        $worksheet->set_column(2, 2, 20);
        $worksheet->write(1, 0,  'Netapp Path', $header);
        $worksheet->write(1, 1,  'Netbackup Policy', $header);
        $worksheet->write(1, 2,  'Status', $header);
        $worksheet->merge_range('A1:F1','Netapp Backup Coverage Report',$format);
}

### Mail Subroutine ###
sub mailit {
        $message = `echo "'Netapp Backup Coverage Report">/tmp/ncr-body.txt`;
        $message = `echo "">>/tmp/ncr-body.txt`;
        $message = `/usr/bin/uuencode /tmp/ncoverage.xls ncoverage.xls > /tmp/ncr-attachment.txt`;
        $message = `cat /tmp/ncr-body.txt /tmp/ncr-attachment.txt > /tmp/ncr.txt`;
        foreach $emails (@emails) {
                $mailit = `/usr/bin/mailx -s $subject $emails < /tmp/ncr.txt`;
        }
}

sub cell {
 #print "NETPATH: $path\n";
 $worksheet->write($num,0,$path);
        $worksheet->write($num,1,$policyname);
        $worksheet->write($num,2,$coverage);
        $num = $num + 1;
}

Netbackup Schedules in 5.x or 6.x

Anyone who has managed Netbackup in a large environment knows how difficult it can be to schedule backup jobs. It is not the creation of the schedule that is the issue, but rather the allocation of resources and knowing when you should schedule a job to run. The issue becomes more cumbersome over time due to new policies being created and older jobs taking longer to run.

I personally faced this challenge myself and found that using the built-in function bpschedreq -predict or nbpemreq -predict was just not adequate. Not only was it cumbersome, I found that some of the time it did not even display jobs that I knew were going to run.

To resolve this issue, I wrote the nbpol script. This script when used will allow you to peer into a specific date and see what schedules will kick off and at what time. It also allows you to print out a histogram summarizing where your volume of jobs run. The syntax is as follows:

Usage: nbpol -y (year) -m (month) -d (day) -t (am|pm|all) (-graph)

Below is the actual perl code:

#!/usr/bin/perl
#########################################################################################
# nbpol: Script to determine which policies will run on a given day when policies #
#  are calendar based.              #
# written: Benjamin Schmaus         #
# date: 070108                      #
#########################################################################################
use DateTime;
use Getopt::Long;
use CGI;
use Time::Local;
use POSIX qw(strftime);
$first = "0";$last = "0";
$schd1 = "SCHED ";$schd2 = "SCHED";
$counter = "0";
$hour = "0";$minute = "0";$second = "0";
$arraycount = 0;
@gac = 0;
options();
@policies = `/usr/openv/netbackup/bin/admincmd/bppllist -l`;
datetime2();
printf "%-30s %-20s %-15s %-15s\n","Policy","Schedule","Start Time","Duration";
printf "%-30s %-20s %-15s %-15s\n","------","--------","----------","--------";
foreach $policies (@policies) {
 $counter = 0;
 chomp($policies);
 open DATA, "/usr/openv/netbackup/bin/admincmd/bpplsched $policies -l|";
 while () {
  $line = $_;
  chomp($line);
  if ($line =~ /$schd1/ && $first eq "0") { first(); }
  if ($line =~ /$schd2/ && $first eq "1") { checks(); }
 }
 close DATA;
 for ($out = 0; $out < $counter; $out++) {
  $policies =~ s/\s*$//g;
  if ($schedcaldayoweek[$out] =~ /$datum/) {
   parseit();
   if ($windowl > 0) {
    if (($time eq "am") && ($starttime < 12)) {
     printf "%-30s %-20s %-15s %-15s\n",$policies,$schedule2,$starttime,$windowl;
     $gac[int($starttime)] = $gac[int($starttime)] + 1;
    } elsif (($time eq "pm") && ($starttime > 11.99)) {
     printf "%-30s %-20s %-15s %-15s\n",$policies,$schedule2,$starttime,$windowl;
     $gac[int($starttime)] = $gac[int($starttime)] + 1;
    } elsif ($time eq "all") {
     printf "%-30s %-20s %-15s %-15s\n",$policies,$schedule2,$starttime,$windowl;
     $gac[int($starttime)] = $gac[int($starttime)] + 1;
    }
   }
  }
 }
}
if ($graph = "1") {
 graphit();
}
exit; 

sub graphit {
 print "\n\n";
 print "Hr\tNumber of Jobs\n";
 print "--\t---------------\n";
 for ($loop = 0; $loop < 24; $loop++) {
  print "$loop\t";
  for ($loop2 = 0; $loop2 < $gac[$loop]; $loop2++) {
   print "*";
  }
  print "\n";
 }
}

sub options {
 $help="";$year="" ;$month="";$day="";$time="";$graph="";
 GetOptions ('h|help'=>\$help,'y|year=s'=>\$year,'m|month=s'=>\$month,'d|day=s'=>\$day,'t|time=s'=>\$time,'graph'=>\$graph);
 if ($help) {
  print "Usage: nbpol -y  -m  -d  -t  [ -graph ]\n";
  exit;
 }
 if (($year eq "") || ($month eq "") || ($day eq "") || ($time eq "")) {
  print "Usage: nbpol -y  -m  -d  -t  [ -graph ]\n";
  exit;
 }
}

sub parseit {
 $field2 = ($dow*2);
 $field1 = ($dow*2)-1;
 @schedtmp = split(/[ \t]+/,$schedule[$out]);
 $schedule2 = $schedtmp[1];
 @schedwintmp = split(/[ \t]+/,$schedwin[$out]);
 $starttime = ($schedwintmp[$field1]/(60*60));
 $starttime =~ s/(^\d{1,}\.\d{2})(.*$)/$1/;
 $windowl = ($schedwintmp[$field2]/(60*60)); 
 $windowl =~s/(^\d{1,}\.\d{2})(.*$)/$1/;
}

sub datetime2 {
 $dt = DateTime->new(year=>$year, month=>$month,day=>$day,hour=>$hour,minute=>$minute,second=>$second,nanosecond=>00,time_zone=>'America/Chicago',);
 print "$dt\n";
 $dow = $dt->day_of_week;      ##### 1-7 (Monday is 1) - also dow, wday
 $wod = $dt->weekday_of_month();  ##### 1-5 weeks
 if ($dow eq "7") { $dow = "1"; } else { $dow = $dow +1; }
 $datum = "$dow,$wod";
 chomp($datum);
}


sub first {
 $first = "1";
 $schedule[$counter] = "$line";
}

sub checks {
 if ($line =~ /SCHEDCALENDAR/) {
  $schedcalendar[$counter] = "SCHEDCALENDAR enabled";
 }
        if ($line =~ /SCHEDCALDAYOWEEK/) {
                $schedcaldayoweek[$counter] = "$line";
        }
        if ($line =~ /SCHEDWIN/) {
                $schedwin[$counter] = "$line";
        }
        if ($line =~ /SCHEDRES/) {
                $schedres[$counter] = "$line";
        }
        if ($line =~ /SCHEDPOOL/) {
                $schedpool[$counter] = "$line";
        }
        if ($line =~ /SCHEDRL/) {
                $schedrl[$counter] = "$line";
        }
        if ($line =~ /SCHEDFOE/) {
                $schedfoe[$counter] = "$line";
  $first = "0";
  $counter = $counter+1;
        }
}

What Perl modules are installed?

Sometimes the package manager of our favorite Linux distribution does not have the Perl modules we need for a development project we are working on.   In that case we have to resort to using CPAN which is analogous to Pip for Python modules.

perl -MCPAN -e shell
cpan> install package-name

Now that process works very well and even prepends dependent modules. However this leads to one pesky problem. How do you know what modules have already been installed and their version?

Well a friend of mine pointed out this gem which is not readily documented on Google:

perl -MCPAN -e 'print CPAN::Shell->r '

This command will tell you what modules and version is installed on the system it is run on.

The other option of course, and I did this at the University of Minnesota, is to download the source Perl modules and make RPM packages. However I find using the CPAN shell to be more convenient.

Perl One Liner to Find Duplicate RPM's in Linux

This little script will use Perl and find duplicate rpms on the system and dump them out to a duplicates file.

rpm --last -qa | perl -n -e '/^(\S+)-\S+-\S+/; print "$&\n" if $SEEN{$1}; $SEEN{$1} ||= $_;' | uniq > duplicates.txt

Perl Script to Restart PVM hosts

Parallel Virtual Machine (PVM) is a software tool for parallel networking of computers. It is designed to allow a network of heterogeneous Unix and/or Windows machines to be used as a single distributed parallel processor.  Thus large computational problems can be solved more cost effectively by using the aggregate power and memory of many computers. The software is very portable; the source code, available free through netlib, has been compiled on everything from laptops to Crays.

The following Perl script will enumerate through the pvmhostfile and restart the pvm on each host.

#!/usr/bin/perl
### This script restarts PVM's that have failed ###
use Parallel::Pvm;
use Net::Ping;
### Set PVM Hostfile location ###
$user="schmaus";
$pvmhostfile="/home/$user/.pvmhostfile";

### Nothing to change below this line ###
open (PVMHOST, $pvmhostfile);
while ($hostname = ) {
chomp($hostname);
$status = Parallel::Pvm::mstat("$hostname");
chomp($status);
if ($status ne "0") {
$alive="1";
$p = Net::Ping->new();
$alive="0" if $p->ping($hostname);
$p->close();
if ($alive ne "0") {
print "$hostname: Offline-Down\n";
} else {
Parallel::Pvm::addhosts("$hostname");
print "$hostname: Offline-Restarted\n";
}
} else {
print "$hostname: Online\n";
}
}
close(PVMHOST);
exit;