OpenStack Cinder: VNX Volume Cleanup

I recently had an issue where I had multiple Cinder volumes in OpenStack (Havana) that were stuck in a deleting state.   Unfortunately the trick of trying to put them back into a state of Available and trying to delete again did not work.    However I was able to come up with a solution to get the job completed and restore consistency.

In my example, my Cinder volumes were being provisioned on a EMC VNX.   So the first step I needed to do was validate if the volumes themselves were removed from the VNX. 

Cleanup on VNX:

1) Obtain the volume ID from either OpenStack Dashboard and/or CLI.
2) Log into Unisphere on the VNX that contains the volume pool where the volumes/luns for Cinder are being provisioned.
3) Select the volume pool and show the luns associated with the volume pool.
4) Filter on the luns using the volume ID obtained in step one.
5) Delete the lun.

Now that we have removed the reference on the VNX, we can continue to do the cleanup on the OpenStack side within the database itself.  This involves editing 3 tables in the cinder mysql database.

1)Obtain the volume ID from either OpenStack Dashboard and/or CLI.   Make note of the volume size as well.   You will also need to obtain the project/tenant ID that the volume belongs to.
2) Login to the OpenStack management controller that runs the MySQL DB.
3) Run the mysql command to access mysql.  Note your deployment may require password and hostname.
4) Select the cinder database with the following syntax:

 mysql>use cinder;

5) Next check if the volume id resides in the volume_admin_metadata table:

mysql> select * from volume_admin_metadata where volume_id="";

6) Delete the volume id if it does:

 mysql> delete from volume_admin_metadata where volume_id="";

7) Next check if the volume id resides in the volumes table:

 mysql> select * from volumes where id="";

8) Delete the volume id if it does:

mysql> delete from volumes where id="";

9) Next update the quota_usages table and reduce the values for the quota_usages fields for that project.  First get a listing to see where things are at:

mysql> select * from quota_usages where project_id="";

10) Then execute the update.  Depending on your setup you will have to update multiple fields from the output in step 9.  In the example below since I was clearing out all volumes for a given project tenant I was able to get away with the following update:

mysql> update quota_usages set in_use='0' where project_id="";

However in cases where you are removing just one volume, you will need to specify the project_id and resource type in the WHERE clause of your MySQL syntax to match on the right in_use field.  Further, your in_use will be either number of GB's  minus volume removed GB's or number of volumes minus one volume removed.

Once I completed this, my system was back in sync and the volumes stuck in Deleting status were gone.

Perl and Isilon's REST API: Part 1 Listing Files In Directory Path

I have been programming in Perl for about 15 years now so anytime I have to write something I usually resort to Perl.   Such was the case when I heard about Isilon and its REST API.   However I soon learned that there is very limited information on Perl and Isilon.

Combing the web I found EMC has a two great manuals that talk about their REST API, the Platform and Namespace manuals (not the official name but provides hints on what to look for).   However these manuals had limited examples and virtually nothing for Perl.   The internet proved to be weak on the subject as well.  I did manage to find some examples with Curl, Python and some read-only type PowerShell scripts.  However I new that whatever I was going to build would at some point need the ability to GET data and PUT/POST data with the REST API.

In the process of my search I did come across a Perl module for Rest::Client.   Seeking out examples of how that module was used with other REST API's and in combination with the EMC manuals for Isilon's REST, I managed to put together my first Perl script.

The following Perl script will list out the files in a directory path on the Isilon cluster:

use REST::Client;
use JSON;
use Data::Dumper;
use MIME::Base64;
use IO::Socket::SSL qw( SSL_VERIFY_NONE );
$ENV{PERL_LWP_SSL_VERIFY_HOSTNAME}=0;
my $username = 'admin';
my $password = 'secret';
my $headers = {Accept => 'application/json', Authorization => 'Basic ' . encode_base64($username . ':' . $password), Content-type => 'application/json'};
my $client = REST::Client->new();
$client->getUseragent()->ssl_opts( SSL_verify_mode => 0 );
$client->setHost('https://10.88.82.106:8080');
$client->GET('/namespace/ifs/data/benz',$headers);
print $client->responseContent(). "\n"; ;
exit;

Here is the output when run from a Windows host:

C:\TEMP>perl isi-file-list.pl
{"children":[{
   "name" : "schmaus3"
}
,{
   "name" : "schmaus1"
}
,{
   "name" : "schmaus2"
}
]}

Basic script that we could then add argument parameters to to fill in the host, username, password and path details.   We could then also write something to parse the JSON response and make it look nice and neat.   Future version, but the core of the example shows you how to do the request in Perl.

Next post will be how you can create a directory on the Isilon cluster.