Security in Kubernetes and OpenShift environments for many customers is very important. Everything from firewall rules, role bindings and limiting access to services is on the table. For customers who run their own private registry using Red Hat Quay that security is extended with Clair which can scan their images as they are placed into the registry for known vulnerabilities. The binary combo of Quay and Clair makes it a great combination. However along with the sense of security comes the need for a smooth transition when upgrading Quay and Clair. In the following blog I will demonstrate how I upgraded my Quay & Clair 3.3.4 environment to 3.5.4.
The upgrade of Quay & Clair from 3.3.4 to 3.5.4 one would think is very trivial but there are a few things that change along the way which means the order of operation is important here. Some of those changes are as follows and ones I experienced:
- From 3.3.4 to 3.4.5 there is a database schema update that is performed so one will want to ensure they have a good backup of their MySQL or PostgreSQL.
- From 3.3.4 to 3.4.5 the Clair scanning version changes from version 2 to version 4. This means we first need to configure a brand new PostgreSQL database and then configure Clair and Quay for v4 scanning.
- From 3.3.4 to 3.4.5 SSL certs now require to have Subject Alternative Names (SAN)
Before we start the upgrade lets look at the current running environment. In all my examples the docker command is used. This could easily be replaced with podman and even in the official documentation podman is the preferred command from Quay 3.4+.
First lets look at the running containers by running the docker ps command:
# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 713cb9e63813 quay.io/redhat/clair-jwt:v3.3.4 "/clair/clair-entr..." 3 hours ago Up 3 hours 0.0.0.0:6060-6061->6060-6061/tcp youthful_ptolemy 32c922276bf4 quay.io/redhat/quay:v3.3.4 "/quay-registry/qu..." 3 hours ago Up 3 hours 7443/tcp, 9091/tcp, 0.0.0.0:80->8080/tcp, 0.0.0.0:443->8443/tcp blissful_thompson 1b05a8997caa registry.access.redhat.com/rhscl/mysql-57-rhel7 "container-entrypo..." 3 hours ago Up 3 hours 0.0.0.0:3306->3306/tcp mysql 9de27908c351 postgres "docker-entrypoint..." 3 hours ago Up 3 hours 0.0.0.0:5432->5432/tcp postgres 53b420f0c0be registry.access.redhat.com/rhscl/redis-32-rhel7 "container-entrypo..." 3 hours ago Up 3 hours 0.0.0.0:6379->6379/tcp jovial_sinoussi
From the output above we can see that in this environment all processes run on the same server with the exception of the S3 storage that actually stores our registry data. So lets break down what is running:
Next I want to show the firewalld rules we have allowed. Note that the list below also shows the additional ports I want opened for the upgrade process.
# firewall-cmd --list-all public (active) target: default icmp-block-inversion: no interfaces: eth0 sources: services: dhcpv6-client ssh ports: 8443/tcp 80/tcp 443/tcp 3306/tcp 6379/tcp 5432/tcp 6060/tcp 6161/tcp 6061/tcp 6062/tcp 6063/tcp 5433/tcp 8080/tcp 8081/tcp 8089/tcp protocols: masquerade: no forward-ports: source-ports: icmp-blocks: rich rules:
Now lets take a look at the current Quay config.yaml file:
# cat /mnt/quay/config/config.yaml
AUTHENTICATION_TYPE: Database
BITTORRENT_FILENAME_PEPPER: af936296-6a1f-444a-8642-a2cd26184cb4
BUILDLOGS_REDIS:
host: quay.schmaustech.com
port: 6379
DATABASE_SECRET_KEY: '19413931088941115129071567106986409590936979178495908481080155841238934557252'
DB_URI: mysql+pymysql://quayuser:JzxCTamgFBmHRhcGFtoPHFkrx1BH2vwQ@192.168.0.11/enterpriseregistrydb
DEFAULT_TAG_EXPIRATION: 2w
DISTRIBUTED_STORAGE_CONFIG:
default:
- RadosGWStorage
- access_key: BKIKJAA5BMMU2RHO6IBB
bucket_name: schmaustech
hostname: 192.168.0.21
is_secure: false
port: '9100'
secret_key: V7f1CwQqAcwo80UEIJEjc5gVQUSSx5ohQ9GSrr12
storage_path: /datastorage/registry
DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS: []
DISTRIBUTED_STORAGE_PREFERENCE:
- default
ENTERPRISE_LOGO_URL: /static/img/RH_Logo_Quay_Black_UX-horizontal.svg
FEATURE_ACI_CONVERSION: false
FEATURE_ANONYMOUS_ACCESS: true
FEATURE_APP_REGISTRY: false
FEATURE_APP_SPECIFIC_TOKENS: true
FEATURE_BUILD_SUPPORT: false
FEATURE_CHANGE_TAG_EXPIRATION: true
FEATURE_DIRECT_LOGIN: true
FEATURE_MAILING: false
FEATURE_PARTIAL_USER_AUTOCOMPLETE: true
FEATURE_REPO_MIRROR: false
FEATURE_REQUIRE_TEAM_INVITE: true
FEATURE_RESTRICTED_V1_PUSH: true
FEATURE_SECURITY_NOTIFICATIONS: true
FEATURE_SECURITY_SCANNER: true
FEATURE_USERNAME_CONFIRMATION: true
FEATURE_USER_CREATION: true
FEATURE_USER_LOG_ACCESS: true
GITHUB_LOGIN_CONFIG: {}
GITHUB_TRIGGER_CONFIG: {}
GITLAB_TRIGGER_KIND: {}
GPG2_PRIVATE_KEY_FILENAME: signing-private.gpg
GPG2_PUBLIC_KEY_FILENAME: signing-public.gpg
LOGS_MODEL: database
LOGS_MODEL_CONFIG: {}
LOG_ARCHIVE_LOCATION: default
MAIL_DEFAULT_SENDER: support@quay.io
MAIL_PORT: 587
MAIL_USE_TLS: true
PREFERRED_URL_SCHEME: https
REGISTRY_TITLE: Red Hat Quay
REGISTRY_TITLE_SHORT: Red Hat Quay
REPO_MIRROR_SERVER_HOSTNAME: null
REPO_MIRROR_TLS_VERIFY: true
SECRET_KEY: '61665332226336867411689064816174816874267671771605676993213345229382029684425'
SECURITY_SCANNER_ENDPOINT: http://quay.schmaustech.com:6060
SECURITY_SCANNER_ISSUER_NAME: security_scanner
SERVER_HOSTNAME: quay.schmaustech.com
SETUP_COMPLETE: true
SIGNING_ENGINE: gpg2
SUPER_USERS:
- admin
TAG_EXPIRATION_OPTIONS:
- 0s
- 1d
- 1w
- 2w
- 4w
TEAM_RESYNC_STALE_TIME: 60m
TESTING: false
USERFILES_LOCATION: default
USERFILES_PATH: userfiles/
USER_EVENTS_REDIS:
host: quay.schmaustech.com
port: 6379
USE_CDN: false
And then lets take a look at the current Clair config.yaml keeping in mind this is Clair v2 and during the upgrade process to Quay 3.4.3 we will be upgrading to Clair v4 so this file will be replaced:
# cat /mnt/clair/config/config.yaml
clair:
database:
type: pgsql
options:
# A PostgreSQL Connection string pointing to the Clair Postgres database.
# Documentation on the format can be found at: http://www.postgresql.org/docs/9.4/static/libpq-connect.html
source: postgresql://postgres:password@quay.schmaustech.com:5432/clairtest?sslmode=disable
cachesize: 16384
api:
#The port at which Clair will report its health status. For example, if Clair is running at
#https://clair.mycompany.com, the health will be reported at
#http://clair.mycompany.com:6061/health.
healthport: 6061
port: 6062
timeout: 900s
# paginationkey can be any random set of characters. *Must be the same across all Clair instances*.
paginationkey:
updater:
# interval defines how often Clair will check for updates from its upstream vulnerability databases.
interval: 6h
notifier:
attempts: 3
renotifyinterval: 1h
http:
# QUAY_ENDPOINT defines the endpoint at which Quay is running.
# For example: https://myregistry.mycompany.com
endpoint: https://quay.schmaustech.com/secscan/notify
proxy: http://localhost:6063
jwtproxy:
signer_proxy:
enabled: true
listen_addr: :6063
ca_key_file: /certificates/mitm.key # Generated internally, do not change.
ca_crt_file: /certificates/mitm.crt # Generated internally, do not change.
signer:
issuer: security_scanner
expiration_time: 5m
max_skew: 1m
nonce_length: 32
private_key:
type: autogenerated
options:
rotate_every: 12h
key_folder: /clair/config/
key_server:
type: keyregistry
options:
# QUAY_ENDPOINT defines the endpoint at which Quay is running.
# For example: https://myregistry.mycompany.com
registry: https://quay.schmaustech.com/keys/
verifier_proxies:
- enabled: true
# The port at which Clair will listen.
listen_addr: :6060
# If Clair is to be served via TLS, uncomment these lines. See the "Running Clair under TLS"
# section below for more information.
#key_file: /clair/config/domain.key
#crt_file: /clair/config/domain.crt
verifier:
# CLAIR_ENDPOINT is the endpoint at which this Clair will be accessible. Note that the port
# specified here must match the listen_addr port a few lines above this.
# Example: https://myclair.mycompany.com:6060
audience: http://quay.schmaustech.com:6060
upstream: http://localhost:6062
key_server:
type: keyregistry
options:
# QUAY_ENDPOINT defines the endpoint at which Quay is running.
# Example: https://myregistry.mycompany.com
registry: https://quay.schmaustech.com/keys/
At this point we have looked at the current running environment but before we begin upgrade I would recommend taking a Quay database backup whether its running MySQL or PostgreSQL. We should do this because during the upgrade process of Quay the DB schema will get updated and so if we had to revert back we would need to restore from backup.
Now lets start the upgrade process from 3.3.4 to 3.4.5. During this upgrade process we will be upgrading three components: Redis, Clair & Quay. Lets start with the Redis upgrade first as we can do this with the current Quay environment. Below we will stop the current Redis container and then start the new Redis container for the 3.4.3 environment:
# docker stop 53b420f0c0be 53b420f0c0be # docker run -d --rm --name redis -p 6379:6379 -e REDIS_PASSWORD=strongpassword registry.redhat.io/rhel8/redis-5:1 84f42084adae1e2e9d5b11aa8e06b90dde49dce015b8f30e72541631c8e98e05 # docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 84f42084adae registry.redhat.io/rhel8/redis-5:1 "container-entrypo..." 5 seconds ago Up 4 seconds 0.0.0.0:6379->6379/tcp redis
Since we added a password switch to Redis on the new version we need to also update the Quay config.yaml to include that password:
BUILDLOGS_REDIS:
host: quay.schmaustech.com
password: strongpassword
port: 6379
USER_EVENTS_REDIS:
host: quay.schmaustech.com
password: strongpassword
port: 6379
Now lets stop and start the Quay 3.3.4 container and confirm our Redis change takes effect:
# docker stop 32c922276bf4 32c922276bf4 # docker run --restart=always -p 443:8443 -p 80:8080 --sysctl net.core.somaxconn=4096 --privileged=true -v /mnt/quay/config:/conf/stack:Z -v /mnt/quay/storage:/datastorage:Z -d quay.io/redhat/quay:v3.3.4 f0ab641a5d253532e263d38820ca1849f17428e0d7d12b6f79b676222d310b06 # docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES f0ab641a5d25 quay.io/redhat/quay:v3.3.4 "/quay-registry/qu..." About a minute ago Up About a minute 7443/tcp, 9091/tcp, 0.0.0.0:80->8080/tcp, 0.0.0.0:443->8443/tcp gifted_einstein
Now lets move onto the upgrade of the Clair component. For this we will need a new database for Clair to consume. In my example I am going to run a new PostgreSQL database container completely separate from the current PostgreSQL container we originally had for our 3.3.4 environment. To do this I will run the following commands:
# mkdir /mnt/postgres-clair4 # setfacl -m u:26:-wx /mnt/postgres-clair4 # docker run -d --rm --name postgresql-clair4 -e POSTGRESQL_USER=clairuser -e POSTGRESQL_PASSWORD=clairpass -e POSTGRESQL_DATABASE=clair -e POSTGRESQL_ADMIN_PASSWORD=password -p 5433:5432 -v /mnt/postgres-clair4:/var/lib/pgsql/data:Z -d registry.redhat.io/rhel8/postgresql-10:1 2b4b1afa3345d24bb6058c713e2f4d363a09da5bff75242d18bad69ddb720d34 # docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 2b4b1afa3345 registry.redhat.io/rhel8/postgresql-10:1 "container-entrypo..." 3 seconds ago Up 2 seconds 0.0.0.0:5433->5432/tcp postgresql-clair4
Because Clair requires the uuid-ossp extensions in the PostgreSQL database we need to run the following SQL command against the database:
# docker exec -it postgresql-clair4 /bin/bash -c 'echo "CREATE EXTENSION IF NOT EXISTS \"uuid-ossp\"" | psql -d clair -U postgres' CREATE EXTENSION
With the Clair v4 PostgreSQL database running lets turn our attention to the Clair v4 config.yaml. Below I created my Clair config path and then added the contents to the config.yaml:
# mkdir -p /mnt/clair4/config
# cat /mnt/clair4/config/config.yaml
http_listen_addr: :8081
introspection_addr: :8089
log_level: debug
indexer:
connstring: host=quay.schmaustech.com port=5433 dbname=clair user=clairuser password=clairpass sslmode=disable
scanlock_retry: 10
layer_scan_concurrency: 5
migrations: true
matcher:
connstring: host=quay.schmaustech.com port=5433 dbname=clair user=clairuser password=clairpass sslmode=disable
max_conn_pool: 100
run: ""
migrations: true
indexer_addr: clair-indexer
notifier:
connstring: host=quay.schmaustech.com port=5433 dbname=clair user=clairuser password=clairpass sslmode=disable
delivery_interval: 1m
poll_interval: 5m
migrations: true
auth:
psk:
key: "MTU5YzA4Y2ZkNzJoMQ=="
iss: ["quay"]
# tracing and metrics
trace:
name: "jaeger"
probability: 1
jaeger:
agent_endpoint: "localhost:6831"
service_name: "clair"
metrics:
name: "prometheus"
Before we start up the Clair v4 container though lets go ahead and make the changes we need to the Quay config.yaml as well. Add or verify the following lines in the configuration:
FEATURE_SECURITY_NOTIFICATIONS: false FEATURE_SECURITY_SCANNER: true SECURITY_SCANNER_INDEXING_INTERVAL: 30 SECURITY_SCANNER_V4_ENDPOINT: http://quay.schmaustech.com:8081 SECURITY_SCANNER_V4_PSK: MTU5YzA4Y2ZkNzJoMQ== SERVER_HOSTNAME: quay.schmaustech.com
At this point we are ready to stop the current Quay and Clair containers:
# docker stop 713cb9e63813 f0ab641a5d25 9de27908c351 713cb9e63813 f0ab641a5d25 9de27908c351 # docker rm 713cb9e63813 f0ab641a5d25 9de27908c351 713cb9e63813 f0ab641a5d25 9de27908c351
Next lets start up the Clair v4 container:
# docker run -d --rm --name clairv4 -p 8081:8081 -p 8089:8089 -e CLAIR_CONF=/clair/config.yaml -e CLAIR_MODE=combo -v /mnt/clair4/config:/clair:Z registry.redhat.io/quay/clair-rhel8:v3.4.5
b627ef1ee4669341d8a5fe72c095e14bb8931200a08d016f95faaec19d466423
[root@quay config]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b627ef1ee466 registry.redhat.io/quay/clair-rhel8:v3.4.5 "/usr/bin/dumb-ini..." 3 seconds ago Up 1 second 0.0.0.0:8081->8081/tcp, 6060/tcp, 0.0.0.0:8089->8089/tcp clairv4And finally lets start up the new Quay 3.4.3 container:
Once Clair v4 is up and running we can then start Quay up on version 3.4.5. Note however that the image scanning may need some additional time to come up as Clair is repopulating the database of vulnerabilities.
# docker run -d --rm -p 443:8443 -p 8080:8080 --name quay -v /mnt/quay/config:/conf/stack:Z -v /mnt/quay/storage:/datastorage:Z registry.redhat.io/quay/quay-rhel8:v3.4.5 1f127bd3791e600a9bcf6f6e20f1fb1e368de27868da802f9ec8d6e5bf87310a # docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 1f127bd3791e registry.redhat.io/quay/quay-rhel8:v3.4.5 "dumb-init -- /qua..." 4 seconds ago Up 4 seconds 7443/tcp, 0.0.0.0:8080->8080/tcp, 0.0.0.0:443->8443/tcp quay
If everything went well we should now have a Quay 3.4.3 and Clair 3.4.5 environment. Before I proceed I want to validate that I can still push/pull content from my registry and that Clair v4 is scanning images appropriately.
Also this is a good time to quickly look at what the running environment should look like:
# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 1f127bd3791e registry.redhat.io/quay/quay-rhel8:v3.4.5 "dumb-init -- /qua..." 35 minutes ago Up 35 minutes 7443/tcp, 0.0.0.0:8080->8080/tcp, 0.0.0.0:443->8443/tcp quay 627ef1ee466 registry.redhat.io/quay/clair-rhel8:v3.4.5 "/usr/bin/dumb-ini..." 38 minutes ago Up 38 minutes 0.0.0.0:8081->8081/tcp, 6060/tcp, 0.0.0.0:8089->8089/tcp clairv4 2b4b1afa3345 registry.redhat.io/rhel8/postgresql-10:1 "container-entrypo..." 2 hours ago Up 2 hours 0.0.0.0:5433->5432/tcp postgresql-clair4 84f42084adae registry.redhat.io/rhel8/redis-5:1 "container-entrypo..." 2 hours ago Up 2 hours 0.0.0.0:6379->6379/tcp redis 1b05a8997caa registry.access.redhat.com/rhscl/mysql-57-rhel7 "container-entrypo..." 20 hours ago Up 20 hours 0.0.0.0:3306->3306/tcp mysql
Here is the chart view again of the environment breakdown:
At this point I am feeling confident we can move forward with the upgrade process to Quay 3.5.4. Thankfully this step is not as complex as the move from 3.3.4 to 3.4.5. In fact its just a matter of switching out the images being used for the containers. The first step then is to stop the current Quay and Clair containers:
# docker stop f8f4f441a89d 1f127bd3791e f8f4f441a89d 1f127bd3791e
In Quay 3.5.4 Helm & OCI artifacts have been enabled by default however since this is an upgrade we need to add those feature switches to the Quay config.yaml if we want them enabled:
FEATURE_GENERAL_OCI_SUPPORT: true FEATURE_HELM_OCI_SUPPORT: true
Once we have stopped the current running containers for Quay and Clair and we have made any changes we needed to the config.yaml we can begin to start the Clair 3.5.4 container:
# docker run -d --rm --name clairv4 -p 8081:8081 -p 8089:8089 -e CLAIR_CONF=/clair/config.yaml -e CLAIR_MODE=combo -v /mnt/clair4/config:/clair:Z registry.redhat.io/quay/clair-rhel8:v3.5.4 0a046b625f4ef348e43a181db81341772cd690162df5eb34478dc525fabd591b # docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 0a046b625f4e registry.redhat.io/quay/clair-rhel8:v3.5.4 "/usr/bin/dumb-ini..." 3 seconds ago Up 2 seconds 0.0.0.0:8081->8081/tcp, 6060/tcp, 0.0.0.0:8089->8089/tcp clairv4
Once Clair is running we can now start the Quay 3.5.4 container:
# docker run -d --rm -p 443:8443 -p 8080:8080 --name quay -v /mnt/quay/config:/conf/stack:Z -v /mnt/quay/storage:/datastorage:Z registry.redhat.io/quay/quay-rhel8:v3.5.4 264ff68abb49fd53b0a2debeffd9684fb6504fa1558b874baadd271ac7a08225 # docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 264ff68abb49 registry.redhat.io/quay/quay-rhel8:v3.5.4 "dumb-init -- /qua..." 4 seconds ago Up 3 seconds 7443/tcp, 0.0.0.0:8080->8080/tcp, 0.0.0.0:443->8443/tcp quay
At this point we should again confirm that we can push/pull images from the Quay environment and also confirm that any new images are being scanned properly by Clair.
If everything checks out and works then we can declare success!
