In my example, my Cinder volumes were being provisioned on a EMC VNX. So the first step I needed to do was validate if the volumes themselves were removed from the VNX.
Cleanup on VNX:
1) Obtain the volume ID from either OpenStack Dashboard and/or CLI.
2) Log into Unisphere on the VNX that contains the volume pool where the volumes/luns for Cinder are being provisioned.
3) Select the volume pool and show the luns associated with the volume pool.
4) Filter on the luns using the volume ID obtained in step one.
5) Delete the lun.
Now that we have removed the reference on the VNX, we can continue to do the cleanup on the OpenStack side within the database itself. This involves editing 3 tables in the cinder mysql database.
1)Obtain the volume ID from either OpenStack Dashboard and/or CLI. Make note of the volume size as well. You will also need to obtain the project/tenant ID that the volume belongs to.
2) Login to the OpenStack management controller that runs the MySQL DB.
3) Run the mysql command to access mysql. Note your deployment may require password and hostname.
4) Select the cinder database with the following syntax:
mysql>use cinder;
5) Next check if the volume id resides in the volume_admin_metadata table:
mysql> select * from volume_admin_metadata where volume_id="
6) Delete the volume id if it does:
mysql> delete from volume_admin_metadata where volume_id="
7) Next check if the volume id resides in the volumes table:
mysql> select * from volumes where id="
8) Delete the volume id if it does:
mysql> delete from volumes where id="
9) Next update the quota_usages table and reduce the values for the quota_usages fields for that project. First get a listing to see where things are at:
mysql> select * from quota_usages where project_id="";
10) Then execute the update. Depending on your setup you will have to update multiple fields from the output in step 9. In the example below since I was clearing out all volumes for a given project tenant I was able to get away with the following update:
mysql> update quota_usages set in_use='0' where project_id="";
However in cases where you are removing just one volume, you will need to specify the project_id and resource type in the WHERE clause of your MySQL syntax to match on the right in_use field. Further, your in_use will be either number of GB's minus volume removed GB's or number of volumes minus one volume removed.
Once I completed this, my system was back in sync and the volumes stuck in Deleting status were gone.
