Saturday, May 21, 2022

Check For Expired Certificates on OpenShift


OpenShift has a lot of certificates associated to the services it runs.  With that in mind it might make sense to check on those certificates every once and a while with some kind of simple report.   I have had customers make this request on occasion and it got me thinking about a quick and dirty way to visualize this.   The following blog show the fruits of this simple task in a simple bash script.

First lets go ahead and create the certs-expired.sh script: 

$ cat << EOF > ~/certs-expired.sh
#!/bin/bash

format="%-8s%-8s%-60s%-26s%-60s\n"
printf "$format" STATE DAYS NAME EXPIRY NAMESPACE
printf "$format" ----- ---- ---- ------ ---------

oc get secrets -A -o go-template='{{range .items}}{{if eq .type "kubernetes.io/tls"}}{{.metadata.namespace}}{{" "}}{{.metadata.name}}{{" "}}{{index .data "tls.crt"}}{{"\n"}}{{end}}{{end}}' | while read namespace name cert
do
  certdate=`echo $cert | base64 -d | openssl x509 -noout -enddate|cut -d= -f2`
  epochcertdate=$(date -d "$certdate" +"%s")
  currentdate=$(date +%s)
  if ((epochcertdate > currentdate)); then
    datediff=$((epochcertdate-currentdate))
    state="OK"
  else
    state="EXPIRED"
    datediff=$((currentdate-epochcertdate))
  fi
  days=$((datediff/86400))
  certdate=`echo $cert | base64 -d | openssl x509 -noout -enddate| cut -d= -f2`
  printf "$format" "$state" "$days" "$name" "$certdate" "$namespace" 
done

EOF

The script makes the assumptions that the oc binary is in the current path variable and that the kubeconfig has been set.   This ensures that the oc command inside the script can pull the appropriate data.   If those assumptions are met we can go ahead and run the script.  I chose just to issue a bash against the script but we could have also set the file with the execute permissions.   When we execute the script we can see the output below:

$ bash certs-expired.sh 
STATE   DAYS    NAME                                                        EXPIRY                    NAMESPACE                                                   
-----   ----    ----                                                        ------                    ---------                                                   
OK      715     openshift-apiserver-operator-serving-cert                   May  5 21:33:47 2024 GMT  openshift-apiserver-operator                                
OK      3635    etcd-client                                                 May  3 21:13:54 2032 GMT  openshift-apiserver                                         
OK      715     serving-cert                                                May  5 21:33:52 2024 GMT  openshift-apiserver                                         
OK      715     serving-cert                                                May  5 21:33:59 2024 GMT  openshift-authentication-operator                           
OK      715     v4-0-config-system-serving-cert                             May  5 21:33:49 2024 GMT  openshift-authentication                                    
OK      715     cloud-credential-operator-serving-cert                      May  5 21:33:50 2024 GMT  openshift-cloud-credential-operator                         
OK      715     machine-approver-tls                                        May  5 21:33:48 2024 GMT  openshift-cluster-machine-approver                          
OK      715     node-tuning-operator-tls                                    May  5 21:33:47 2024 GMT  openshift-cluster-node-tuning-operator                      
OK      715     samples-operator-tls                                        May  5 21:37:44 2024 GMT  openshift-cluster-samples-operator                          
OK      715     cluster-storage-operator-serving-cert                       May  5 21:33:55 2024 GMT  openshift-cluster-storage-operator                          
OK      715     csi-snapshot-webhook-secret                                 May  5 21:33:47 2024 GMT  openshift-cluster-storage-operator                          
OK      715     serving-cert                                                May  5 21:33:54 2024 GMT  openshift-cluster-storage-operator                          
OK      715     cluster-version-operator-serving-cert                       May  5 21:33:52 2024 GMT  openshift-cluster-version                                   
OK      15      kube-controller-manager-client-cert-key                     Jun  5 21:33:41 2022 GMT  openshift-config-managed                                    
OK      15      kube-scheduler-client-cert-key                              Jun  5 21:33:34 2022 GMT  openshift-config-managed                                    
OK      715     config-operator-serving-cert                                May  5 21:33:47 2024 GMT  openshift-config-operator                                   
OK      3635    etcd-client                                                 May  3 21:13:54 2032 GMT  openshift-config                                            
OK      3635    etcd-metric-client                                          May  3 21:13:54 2032 GMT  openshift-config                                            
OK      3635    etcd-metric-signer                                          May  3 21:13:54 2032 GMT  openshift-config                                            
OK      3635    etcd-signer                                                 May  3 21:13:54 2032 GMT  openshift-config                                            
OK      715     serving-cert                                                May  5 21:41:37 2024 GMT  openshift-console-operator                                  
OK      715     console-serving-cert                                        May  5 21:42:15 2024 GMT  openshift-console                                           
OK      715     openshift-controller-manager-operator-serving-cert          May  5 21:33:47 2024 GMT  openshift-controller-manager-operator                       
OK      715     serving-cert                                                May  5 21:33:56 2024 GMT  openshift-controller-manager                                
OK      715     metrics-tls                                                 May  5 21:33:58 2024 GMT  openshift-dns-operator                                      
OK      715     dns-default-metrics-tls                                     May  5 21:34:59 2024 GMT  openshift-dns                                               
OK      3635    etcd-client                                                 May  3 21:13:54 2032 GMT  openshift-etcd-operator                                     
OK      715     etcd-operator-serving-cert                                  May  5 21:33:57 2024 GMT  openshift-etcd-operator                                     
OK      3635    etcd-client                                                 May  3 21:13:54 2032 GMT  openshift-etcd                                              
OK      1080    etcd-peer-asus-vm1.kni.schmaustech.com                      May  5 21:51:28 2025 GMT  openshift-etcd                                              
OK      1080    etcd-peer-asus1-vm2.kni.schmaustech.com                     May  5 21:33:23 2025 GMT  openshift-etcd                                              
OK      1080    etcd-peer-asus1-vm3.kni.schmaustech.com                     May  5 21:33:24 2025 GMT  openshift-etcd                                              
OK      1080    etcd-serving-asus-vm1.kni.schmaustech.com                   May  5 21:51:28 2025 GMT  openshift-etcd                                              
OK      1080    etcd-serving-asus1-vm2.kni.schmaustech.com                  May  5 21:33:23 2025 GMT  openshift-etcd                                              
OK      1080    etcd-serving-asus1-vm3.kni.schmaustech.com                  May  5 21:33:24 2025 GMT  openshift-etcd                                              
OK      1080    etcd-serving-metrics-asus-vm1.kni.schmaustech.com           May  5 21:51:27 2025 GMT  openshift-etcd                                              
OK      1080    etcd-serving-metrics-asus1-vm2.kni.schmaustech.com          May  5 21:33:23 2025 GMT  openshift-etcd                                              
OK      1080    etcd-serving-metrics-asus1-vm3.kni.schmaustech.com          May  5 21:33:24 2025 GMT  openshift-etcd                                              
OK      715     serving-cert                                                May  5 21:33:59 2024 GMT  openshift-etcd                                              
OK      715     image-registry-operator-tls                                 May  5 21:33:58 2024 GMT  openshift-image-registry                                    
OK      715     metrics-tls                                                 May  5 21:33:55 2024 GMT  openshift-ingress-operator                                  
OK      715     router-ca                                                   May  5 21:35:59 2024 GMT  openshift-ingress-operator                                  
OK      715     router-certs-default                                        May  5 21:36:01 2024 GMT  openshift-ingress                                           
OK      715     router-metrics-certs-default                                May  5 21:36:00 2024 GMT  openshift-ingress                                           
OK      715     openshift-insights-serving-cert                             May  5 21:33:51 2024 GMT  openshift-insights                                          
OK      15      aggregator-client-signer                                    Jun  6 16:21:59 2022 GMT  openshift-kube-apiserver-operator                           
OK      715     kube-apiserver-operator-serving-cert                        May  5 21:33:54 2024 GMT  openshift-kube-apiserver-operator                           
OK      350     kube-apiserver-to-kubelet-signer                            May  6 21:09:57 2023 GMT  openshift-kube-apiserver-operator                           
OK      350     kube-control-plane-signer                                   May  6 21:09:57 2023 GMT  openshift-kube-apiserver-operator                           
OK      3635    loadbalancer-serving-signer                                 May  3 21:09:52 2032 GMT  openshift-kube-apiserver-operator                           
OK      3635    localhost-recovery-serving-signer                           May  3 21:33:29 2032 GMT  openshift-kube-apiserver-operator                           
OK      3635    localhost-serving-signer                                    May  3 21:09:50 2032 GMT  openshift-kube-apiserver-operator                           
OK      105     node-system-admin-client                                    Sep  3 21:33:40 2022 GMT  openshift-kube-apiserver-operator                           
OK      350     node-system-admin-signer                                    May  6 21:33:29 2023 GMT  openshift-kube-apiserver-operator                           
OK      3635    service-network-serving-signer                              May  3 21:09:51 2032 GMT  openshift-kube-apiserver-operator                           
OK      15      aggregator-client                                           Jun  6 16:21:59 2022 GMT  openshift-kube-apiserver                                    
OK      15      check-endpoints-client-cert-key                             Jun  5 21:33:46 2022 GMT  openshift-kube-apiserver                                    
OK      15      control-plane-node-admin-client-cert-key                    Jun  5 21:33:53 2022 GMT  openshift-kube-apiserver                                    
OK      3635    etcd-client                                                 May  3 21:13:54 2032 GMT  openshift-kube-apiserver                                    
OK      3635    etcd-client-10                                              May  3 21:13:54 2032 GMT  openshift-kube-apiserver                                    
OK      3635    etcd-client-11                                              May  3 21:13:54 2032 GMT  openshift-kube-apiserver                                    
OK      3635    etcd-client-12                                              May  3 21:13:54 2032 GMT  openshift-kube-apiserver                                    
OK      3635    etcd-client-8                                               May  3 21:13:54 2032 GMT  openshift-kube-apiserver                                    
OK      3635    etcd-client-9                                               May  3 21:13:54 2032 GMT  openshift-kube-apiserver                                    
OK      15      external-loadbalancer-serving-certkey                       Jun  5 21:33:52 2022 GMT  openshift-kube-apiserver                                    
OK      15      internal-loadbalancer-serving-certkey                       Jun  5 21:33:34 2022 GMT  openshift-kube-apiserver                                    
OK      15      kubelet-client                                              Jun  5 21:33:34 2022 GMT  openshift-kube-apiserver                                    
OK      3635    localhost-recovery-serving-certkey                          May  3 21:33:29 2032 GMT  openshift-kube-apiserver                                    
OK      3635    localhost-recovery-serving-certkey-10                       May  3 21:33:29 2032 GMT  openshift-kube-apiserver                                    
OK      3635    localhost-recovery-serving-certkey-11                       May  3 21:33:29 2032 GMT  openshift-kube-apiserver                                    
OK      3635    localhost-recovery-serving-certkey-12                       May  3 21:33:29 2032 GMT  openshift-kube-apiserver                                    
OK      3635    localhost-recovery-serving-certkey-8                        May  3 21:33:29 2032 GMT  openshift-kube-apiserver                                    
OK      3635    localhost-recovery-serving-certkey-9                        May  3 21:33:29 2032 GMT  openshift-kube-apiserver                                    
OK      15      localhost-serving-cert-certkey                              Jun  5 21:33:34 2022 GMT  openshift-kube-apiserver                                    
OK      15      service-network-serving-certkey                             Jun  5 21:33:33 2022 GMT  openshift-kube-apiserver                                    
OK      15      csr-signer                                                  Jun  6 16:26:40 2022 GMT  openshift-kube-controller-manager-operator                  
OK      45      csr-signer-signer                                           Jul  6 16:22:14 2022 GMT  openshift-kube-controller-manager-operator                  
OK      715     kube-controller-manager-operator-serving-cert               May  5 21:33:57 2024 GMT  openshift-kube-controller-manager-operator                  
OK      15      csr-signer                                                  Jun  6 16:26:40 2022 GMT  openshift-kube-controller-manager                           
OK      15      kube-controller-manager-client-cert-key                     Jun  5 21:33:41 2022 GMT  openshift-kube-controller-manager                           
OK      715     serving-cert                                                May  5 21:33:51 2024 GMT  openshift-kube-controller-manager                           
OK      715     serving-cert-2                                              May  5 21:33:51 2024 GMT  openshift-kube-controller-manager                           
OK      715     serving-cert-3                                              May  5 21:33:51 2024 GMT  openshift-kube-controller-manager                           
OK      715     serving-cert-4                                              May  5 21:33:51 2024 GMT  openshift-kube-controller-manager                           
OK      715     serving-cert-5                                              May  5 21:33:51 2024 GMT  openshift-kube-controller-manager                           
OK      715     serving-cert-6                                              May  5 21:33:51 2024 GMT  openshift-kube-controller-manager                           
OK      715     serving-cert-7                                              May  5 21:33:51 2024 GMT  openshift-kube-controller-manager                           
OK      715     kube-scheduler-operator-serving-cert                        May  5 21:33:50 2024 GMT  openshift-kube-scheduler-operator                           
OK      15      kube-scheduler-client-cert-key                              Jun  5 21:33:34 2022 GMT  openshift-kube-scheduler                                    
OK      715     serving-cert                                                May  5 21:33:59 2024 GMT  openshift-kube-scheduler                                    
OK      715     serving-cert-3                                              May  5 21:33:59 2024 GMT  openshift-kube-scheduler                                    
OK      715     serving-cert-4                                              May  5 21:33:59 2024 GMT  openshift-kube-scheduler                                    
OK      715     serving-cert-5                                              May  5 21:33:59 2024 GMT  openshift-kube-scheduler                                    
OK      715     serving-cert-6                                              May  5 21:33:59 2024 GMT  openshift-kube-scheduler                                    
OK      715     serving-cert-7                                              May  5 21:33:59 2024 GMT  openshift-kube-scheduler                                    
OK      715     serving-cert                                                May  5 21:34:00 2024 GMT  openshift-kube-storage-version-migrator-operator            
OK      725     diskmaker-metric-serving-cert                               May 15 23:33:46 2024 GMT  openshift-local-storage                                     
OK      715     baremetal-operator-webhook-server-cert                      May  5 21:36:34 2024 GMT  openshift-machine-api                                       
OK      715     cluster-autoscaler-operator-cert                            May  5 21:34:01 2024 GMT  openshift-machine-api                                       
OK      715     cluster-baremetal-operator-tls                              May  5 21:33:58 2024 GMT  openshift-machine-api                                       
OK      715     cluster-baremetal-webhook-server-cert                       May  5 21:33:48 2024 GMT  openshift-machine-api                                       
OK      715     machine-api-controllers-tls                                 May  5 21:33:47 2024 GMT  openshift-machine-api                                       
OK      715     machine-api-operator-tls                                    May  5 21:33:56 2024 GMT  openshift-machine-api                                       
OK      715     machine-api-operator-webhook-cert                           May  5 21:33:53 2024 GMT  openshift-machine-api                                       
OK      715     proxy-tls                                                   May  5 21:34:00 2024 GMT  openshift-machine-config-operator                           
OK      715     marketplace-operator-metrics                                May  5 21:33:50 2024 GMT  openshift-marketplace                                       
OK      715     alertmanager-main-tls                                       May  5 21:45:20 2024 GMT  openshift-monitoring                                        
OK      715     cluster-monitoring-operator-tls                             May  5 21:33:52 2024 GMT  openshift-monitoring                                        
OK      715     grafana-tls                                                 May  5 21:45:20 2024 GMT  openshift-monitoring                                        
OK      715     kube-state-metrics-tls                                      May  5 21:35:59 2024 GMT  openshift-monitoring                                        
OK      715     node-exporter-tls                                           May  5 21:35:59 2024 GMT  openshift-monitoring                                        
OK      715     openshift-state-metrics-tls                                 May  5 21:35:58 2024 GMT  openshift-monitoring                                        
OK      715     prometheus-adapter-tls                                      May  5 21:35:59 2024 GMT  openshift-monitoring                                        
OK      715     prometheus-k8s-thanos-sidecar-tls                           May  5 21:45:22 2024 GMT  openshift-monitoring                                        
OK      715     prometheus-k8s-tls                                          May  5 21:45:21 2024 GMT  openshift-monitoring                                        
OK      715     prometheus-operator-tls                                     May  5 21:35:43 2024 GMT  openshift-monitoring                                        
OK      715     telemeter-client-tls                                        May  5 21:37:44 2024 GMT  openshift-monitoring                                        
OK      715     thanos-querier-tls                                          May  5 21:35:58 2024 GMT  openshift-monitoring                                        
OK      715     metrics-daemon-secret                                       May  5 21:33:56 2024 GMT  openshift-multus                                            
OK      715     multus-admission-controller-secret                          May  5 21:33:48 2024 GMT  openshift-multus                                            
OK      3635    etcd-client                                                 May  3 21:13:54 2032 GMT  openshift-oauth-apiserver                                   
OK      715     serving-cert                                                May  5 21:34:01 2024 GMT  openshift-oauth-apiserver                                   
OK      715     catalog-operator-serving-cert                               May  5 21:33:47 2024 GMT  openshift-operator-lifecycle-manager                        
OK      715     olm-operator-serving-cert                                   May  5 21:33:48 2024 GMT  openshift-operator-lifecycle-manager                        
OK      714     packageserver-service-cert                                  May  4 21:34:44 2024 GMT  openshift-operator-lifecycle-manager                        
OK      0       pprof-cert                                                  May 21 18:30:03 2022 GMT  openshift-operator-lifecycle-manager                        
OK      3635    ovn-ca                                                      May  3 21:27:45 2032 GMT  openshift-ovn-kubernetes                                    
OK      167     ovn-cert                                                    Nov  5 09:27:45 2022 GMT  openshift-ovn-kubernetes                                    
OK      715     ovn-master-metrics-cert                                     May  5 21:33:53 2024 GMT  openshift-ovn-kubernetes                                    
OK      715     ovn-node-metrics-cert                                       May  5 21:33:49 2024 GMT  openshift-ovn-kubernetes                                    
OK      3635    signer-ca                                                   May  3 21:27:46 2032 GMT  openshift-ovn-kubernetes                                    
OK      167     signer-cert                                                 Nov  5 09:27:46 2022 GMT  openshift-ovn-kubernetes                                    
OK      715     serving-cert                                                May  5 21:33:54 2024 GMT  openshift-service-ca-operator                               
OK      775     signing-key                                                 Jul  4 21:33:37 2024 GMT  openshift-service-ca                                        
OK      725     noobaa-db-serving-cert                                      May 15 23:42:26 2024 GMT  openshift-storage                                           
OK      725     noobaa-mgmt-serving-cert                                    May 15 23:42:26 2024 GMT  openshift-storage                                           
OK      725     noobaa-operator-service-cert                                May 16 06:23:29 2024 GMT  openshift-storage                                           
OK      725     noobaa-s3-serving-cert                                      May 15 23:42:26 2024 GMT  openshift-storage                                           
OK      725     ocs-storagecluster-cos-ceph-rgw-tls-cert                    May 15 23:41:32 2024 GMT  openshift-storage                                           
OK      725     odf-console-serving-cert                                    May 15 23:27:38 2024 GMT  openshift-storage   

The output of the script is simple.  The first column contains the state of the certificate.  If its okay then it just says OK and if its expired the field will say EXPIRED.   The next column tells us how many days until the certificate expires and if the number is negative then the certificate is expired and has been for that many days.   The third column tells us the certificates name while the fourth gives us the actual expiry date.   Finally the last column provides the namespace the certificate is in.

Again just a simple script but provides an example of how we can see this type of information.  However if one has a fleet of clusters then configuring a Red Hat Advanced Cluster Management Certificate Policy Controller might be a more effective method at expired certificate management.