OpenStack Havana - Recovering Services Account When Deleted

I was working with one of my colleagues who had accidentally deleted the services account with OpenStack.   Unfortunately if this happens, it tends to break your setup in a really big way.   After opening a case with Redhat whose OpenStack distribution we were using led to no results.  I managed to reverse engineer where the services account resided and reestablish it.  Here are the steps I did:

Symptoms:

1) In web gui user gets "Oops something went wrong!" when trying to login.   User can get get valid token at command line (keystone token-get) but authorization fails.

2) Openstack-status shows the following:

                == Glance images ==

                Request returned failure status.

                Invalid OpenStack Identity credentials.

                ==Nova managed services ==

                ERROR: Unauthorized (HTTP 401)

                == Nova networks ==

                ERROR: Unauthorized (HTTP 401)

                == Nova instance flavors ==

                ERROR: Unauthorized (HTTP 401)

                == Nova instances ==

                ERROR: Unauthorized (HTTP 401)

Resolution:

Create New Services Project:

Create new "services" project/tenant via CLI (keystone tenant-create).

Obtain new "services" project/tenant ID via CLI (keystone tenant-list).

Determine NEW_SERVICES_ID:

Determine old project/tenant id of services project by looking at following users (nova,glance,neutron,heat,cinder) default_project_id in the user table of keystone database.   There default_project_id should all be the same and was the ID of the previous services project that was removed.

Edit MySQL Database:

use keystone;

update user set default_project_id="NEW_SERVICES_ID" where default_project_id="OLD_SERVICES_ID";

use ovs_neutron;

update networks set tenant_id="NEW_SERVICES_ID" where tenant_id="OLD_SERVICES_ID";

update subnets set tenant_id="NEW_SERVICES_ID" where tenant_id="OLD_SERVICES_ID";

update securitygroups set tenant_id="NEW_SERVICES_ID" where tenant_id="OLD_SERVICES_ID";