Saturday, February 15, 2020
OpenShift 4.3 Baremetal Deployment Troubleshooting Flow
The following is just a minimalist approach to troubleshooting an OpenShift 4.3 baremetal deployment. Obviously it does not cover everything in detail but it still provides an initial guide.
Tuesday, February 04, 2020
Carve up RedFish with Sushy-Tools
The Redfish API is a specification that delivers an industry standard protocol with a RESTful interface for the management of servers, storage, networking and converged infrastructure. Hardware vendors like Dell, HPE, Lenovo and Supermicro have all released servers with Redfish capabilities. However if you are a software engineer like myself that has barriers to getting access to Redfish capable hardware or hardware in general and find yourself often integrating and coding against virtual machines then Sushy tools might be exotic taste you crave.
If you are not familiar with Sushy-tools then you have come to the right place as this blog will show you how to acquire, configure and use Sushy-tools with your KVM virtual machines. The first thing we need to do is get Sushy tools and we have two options: install via Python pip or container. I am opting for using a container and the rest of this document will cover Sushy-tools from that standpoint although much of what we cover would apply to if it were installed with pip as well.
Installation/Configuration:
In order to use the Sushy tool container we first need to install podman:
# yum -y install podman
Next lets create a mount point for the container on our host:
# mkdir /opt/sushy
Now we can export our Sushy tool image to a variable and then create the container with podman:
# export SUSHY_TOOLS_IMAGE=${SUSHY_TOOLS_IMAGE:-"quay.io/metal3-io/sushy-tools"}
# podman create -d --net host --privileged --name sushy-tools -v "/opt/sushy":/root/sushy -v "/root/.ssh":/root/.ssh "${SUSHY_TOOLS_IMAGE}"
6ee305e4a435605ef8f0716878b6f06ad1eb38a12c8c896d779b65a4d1995ad3
Once the container is created we can start it up and confirm with a podman ps that it is running:
# podman start sushy-tools sushy-tools # sudo podman ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 6ee305e4a435 quay.io/metal3-io/sushy-tools:latest /bin/sh -c sushy-... 23 seconds ago Up 4 seconds ago sushy-tools
Lets go ahead and peek inside and see what processes are running at startup. It will look similar to what we see below:
# podman exec -it 6ee305e4a435 /bin/bash # # ps -efl F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD 4 S root 1 0 0 80 0 - 597 - 21:45 ? 00:00:00 /bin/sh -c sushy-emulator -i 0.0.0.0 -p 8000 --config /root/sushy/conf.py 4 S root 6 1 0 80 0 - 26468 x64_sy 21:45 ? 00:00:00 /usr/local/bin/python /usr/local/bin/sushy-emulator -i 0.0.0.0 -p 8000 --config /root/sushy/conf.py 4 S root 7 0 0 80 0 - 1438 - 21:47 pts/0 00:00:00 /bin/bash 0 R root 13 7 0 80 0 - 2348 - 21:48 pts/0 00:00:00 ps -efl
Now I have a libvirt KVM hypervisor host I would like to control the virtual machines on from a Redfish perspective. So the first thing I need to do is copy my public sshkey over to that host:
# ssh-copy-id root@192.168.0.5
Once the key is copied we can start a Sushy emulator process specific for that host with the following syntax. I am specifying a -i and ipaddress to ensure this port is listening so I can run my curl commands from remote hosts. Also note that you will need to open any firewall ports based on the port number specify in order to access the API.
# /usr/local/bin/sushy-emulator -i 192.168.0.11 -p 9005 --libvirt-uri qemu+ssh://root@192.168.0.5/system & [1] 18 # * Serving Flask app "sushy_tools.emulator.main" (lazy loading) * Environment: production WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead. * Debug mode: off * Running on http://192.168.0.11:9005/ (Press CTRL+C to quit) # ps -efl F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD 4 S root 1 0 0 80 0 - 597 - 21:54 ? 00:00:00 /bin/sh -c sushy-emulator -i 0.0.0.0 -p 8000 --config /root/sushy/conf.py 4 S root 6 1 0 80 0 - 44994 x64_sy 21:54 ? 00:00:00 /usr/local/bin/python /usr/local/bin/sushy-emulator -i 0.0.0.0 -p 8000 --config /root/sushy/conf.py 4 S root 10 0 0 80 0 - 1438 - 22:07 pts/0 00:00:00 /bin/bash 4 S root 18 10 0 80 0 - 26492 x64_sy 22:07 pts/0 00:00:00 /usr/local/bin/python /usr/local/bin/sushy-emulator -i 192.168.0.11 -p 9005 --libvirt-uri qemu+ssh://root@192.168.0.5/system 0 R root 19 10 0 80 0 - 2348 - 22:08 pts/0 00:00:00 ps -efl
After confirming our process is up and running we can move on to using Sushy tools to explore what virtual machines are on the hypervisor we specified in our previous command.
Using the Tools:
As stated before Sushy tools provides a Redfish emulated API for libvirt KVM. In the discussion above we successfully installed and configured Sushy tools and enabled them to access a hypervisor host at the address of 192.168.0.5. Now lets see what kind of information the Sushy tools will provide us.
If we run a curl command to retrieve the virtual machines on the host we can see there are currently 3 of them:
# curl 192.168.0.11:9005/redfish/v1/Systems
{
"@odata.type": "#ComputerSystemCollection.ComputerSystemCollection",
"Name": "Computer System Collection",
"Members@odata.count": 3,
"Members": [
{
"@odata.id": "/redfish/v1/Systems/021359d3-9335-4e0c-800c-ad40232d804b"
},
{
"@odata.id": "/redfish/v1/Systems/04383b48-82c5-446a-a2db-5555b265ce75"
},
{
"@odata.id": "/redfish/v1/Systems/668d623c-2453-4c09-ac7d-8e509e91a337"
}
],
"@odata.context": "/redfish/v1/$metadata#ComputerSystemCollection.ComputerSystemCollection",
"@odata.id": "/redfish/v1/Systems",
"@Redfish.Copyright": "Copyright 2014-2016 Distributed Management Task Force, Inc. (DMTF). For the full DMTF copyright policy, see http://www.dmtf.org/about/policies/copyright."
}
Now lets explore the first system a little further and see what state it is in:
# curl 192.168.0.11:9005/redfish/v1/Systems/021359d3-9335-4e0c-800c-ad40232d804b
{
"@odata.type": "#ComputerSystem.v1_1_0.ComputerSystem",
"Id": "021359d3-9335-4e0c-800c-ad40232d804b",
"Name": "Jenkins",
"UUID": "021359d3-9335-4e0c-800c-ad40232d804b",
"Status": {
"State": "Enabled",
"Health": "OK",
"HealthRollUp": "OK"
},
"PowerState": "Off",
"Boot": {
"BootSourceOverrideEnabled": "Continuous"
},
"ProcessorSummary": {
"Count": 4,
"Status": {
"State": "Enabled",
"Health": "OK",
"HealthRollUp": "OK"
}
},
"MemorySummary": {
"TotalSystemMemoryGiB": 14,
"Status": {
"State": "Enabled",
"Health": "OK",
"HealthRollUp": "OK"
}
},
"Bios": {
"@odata.id": "/redfish/v1/Systems/021359d3-9335-4e0c-800c-ad40232d804b/BIOS"
},
"Processors": {
"@odata.id": "/redfish/v1/Systems/021359d3-9335-4e0c-800c-ad40232d804b/Processors"
},
"Memory": {
"@odata.id": "/redfish/v1/Systems/021359d3-9335-4e0c-800c-ad40232d804b/Memory"
},
"EthernetInterfaces": {
"@odata.id": "/redfish/v1/Systems/021359d3-9335-4e0c-800c-ad40232d804b/EthernetInterfaces"
},
"SimpleStorage": {
"@odata.id": "/redfish/v1/Systems/021359d3-9335-4e0c-800c-ad40232d804b/SimpleStorage"
},
"Storage": {
"@odata.id": "/redfish/v1/Systems/021359d3-9335-4e0c-800c-ad40232d804b/Storage"
},
"IndicatorLED": "Lit",
"Links": {
"Chassis": [
{
"@odata.id": "/redfish/v1/Chassis/15693887-7984-9484-3272-842188918912"
}
],
"ManagedBy": [
{
"@odata.id": "/redfish/v1/Managers/58893887-8974-2487-2389-841168418919"
}
]
},
"Actions": {
"#ComputerSystem.Reset": {
"target": "/redfish/v1/Systems/021359d3-9335-4e0c-800c-ad40232d804b/Actions/ComputerSystem.Reset",
"ResetType@Redfish.AllowableValues": [
"On",
"ForceOff",
"GracefulShutdown",
"GracefulRestart",
"ForceRestart",
"Nmi",
"ForceOn"
]
}
},
"@odata.context": "/redfish/v1/$metadata#ComputerSystem.ComputerSystem",
"@odata.id": "/redfish/v1/Systems/021359d3-9335-4e0c-800c-ad40232d804b",
"@Redfish.Copyright": "Copyright 2014-2016 Distributed Management Task Force, Inc. (DMTF). For the full DMTF copyright policy, see http://www.dmtf.org/about/policies/copyright."
}
We can see from the results we can learn quite a bit about the virtual machine like its name, status, power state, memory and cpu. I noticed this system was off and it appears to be the Jenkins CI/CD host. So lets show how Sushy tools can be useful and turn this machine back on:
# curl -d '{"ResetType":"On"}' -H "Content-Type: application/json" -X POST http://192.168.0.11:9005/redfish/v1/Systems/021359d3-9335-4e0c-800c-ad40232d804b/Actions/ComputerSystem.Reset
After a few seconds lets confirm its on by running the following command we ran earlier:
# curl 192.168.0.11:9005/redfish/v1/Systems/021359d3-9335-4e0c-800c-ad40232d804b
{
"@odata.type": "#ComputerSystem.v1_1_0.ComputerSystem",
"Id": "021359d3-9335-4e0c-800c-ad40232d804b",
"Name": "Jenkins",
"UUID": "021359d3-9335-4e0c-800c-ad40232d804b",
"Status": {
"State": "Enabled",
"Health": "OK",
"HealthRollUp": "OK"
},
"PowerState": "On",
"Boot": {
"BootSourceOverrideEnabled": "Continuous"
},
"ProcessorSummary": {
"Count": 4,
"Status": {
"State": "Enabled",
"Health": "OK",
"HealthRollUp": "OK"
}
},
"MemorySummary": {
"TotalSystemMemoryGiB": 14,
"Status": {
"State": "Enabled",
"Health": "OK",
"HealthRollUp": "OK"
}
},
"Bios": {
"@odata.id": "/redfish/v1/Systems/021359d3-9335-4e0c-800c-ad40232d804b/BIOS"
},
"Processors": {
"@odata.id": "/redfish/v1/Systems/021359d3-9335-4e0c-800c-ad40232d804b/Processors"
},
"Memory": {
"@odata.id": "/redfish/v1/Systems/021359d3-9335-4e0c-800c-ad40232d804b/Memory"
},
"EthernetInterfaces": {
"@odata.id": "/redfish/v1/Systems/021359d3-9335-4e0c-800c-ad40232d804b/EthernetInterfaces"
},
"SimpleStorage": {
"@odata.id": "/redfish/v1/Systems/021359d3-9335-4e0c-800c-ad40232d804b/SimpleStorage"
},
"Storage": {
"@odata.id": "/redfish/v1/Systems/021359d3-9335-4e0c-800c-ad40232d804b/Storage"
},
"IndicatorLED": "Lit",
"Links": {
"Chassis": [
{
"@odata.id": "/redfish/v1/Chassis/15693887-7984-9484-3272-842188918912"
}
],
"ManagedBy": [
{
"@odata.id": "/redfish/v1/Managers/58893887-8974-2487-2389-841168418919"
}
]
},
"Actions": {
"#ComputerSystem.Reset": {
"target": "/redfish/v1/Systems/021359d3-9335-4e0c-800c-ad40232d804b/Actions/ComputerSystem.Reset",
"ResetType@Redfish.AllowableValues": [
"On",
"ForceOff",
"GracefulShutdown",
"GracefulRestart",
"ForceRestart",
"Nmi",
"ForceOn"
]
}
},
"@odata.context": "/redfish/v1/$metadata#ComputerSystem.ComputerSystem",
"@odata.id": "/redfish/v1/Systems/021359d3-9335-4e0c-800c-ad40232d804b",
"@Redfish.Copyright": "Copyright 2014-2016 Distributed Management Task Force, Inc. (DMTF). For the full DMTF copyright policy, see http://www.dmtf.org/about/policies/copyright."
}
Sure enough the system is now powered on and running all via the Sushy tools. While we are here lets take a look at the systems storage for fun:
# curl 192.168.0.11:9005/redfish/v1/Systems/021359d3-9335-4e0c-800c-ad40232d804b/SimpleStorage
{
"@odata.type": "#SimpleStorageCollection.SimpleStorageCollection",
"Name": "Simple Storage Collection",
"Members@odata.count": 1,
"Members": [
{
"@odata.id": "/redfish/v1/Systems/021359d3-9335-4e0c-800c-ad40232d804b/SimpleStorage/virtio"
}
],
"Oem": {},
"@odata.context": "/redfish/v1/$metadata#SimpleStorageCollection.SimpleStorageCollection",
"@odata.id": "/redfish/v1/Systems/021359d3-9335-4e0c-800c-ad40232d804b/SimpleStorage"
}
The above command tells us there is one virtio disk. Now if we take the data id of that disk and curl on that we can get even further details about the disk, its capacity and status:
# curl 192.168.0.11:9005/redfish/v1/Systems/021359d3-9335-4e0c-800c-ad40232d804b/SimpleStorage/virtio
{
"@odata.type": "#SimpleStorage.v1_2_0.SimpleStorage",
"Id": "virtio",
"Name": "virtio Controller",
"Devices": [
{
"@odata.type": "#SimpleStorage.v1_1_0.Device",
"Name": "Jenkins.qcow2",
"CapacityBytes": 64424509440,
"Status": {
"@odata.type": "#Resource.Status",
"State": "Enabled",
"Health": "OK"
}
}
],
"@odata.context": "/redfish/v1/$metadata#SimpleStorage.SimpleStorage",
"@odata.id": "/redfish/v1/Systems/021359d3-9335-4e0c-800c-ad40232d804b/SimpleStorage/virtio"
}
As you can see Sushy tools provides a way to enable the Redfish API for libvirt KVM virtual machines. Providing the ability to develop where the Redfish API is required without the need to invest in enterprise hardware.
Subscribe to:
Posts (Atom)