Thursday, July 08, 2021

Simplify Red Hat Quay/Clair Upgrade From 3.3.4 to 3.5.4

 

Security in Kubernetes and OpenShift environments for many customers is very important.  Everything from firewall rules, role bindings and limiting access to services is on the table.   For customers who run their own private registry using Red Hat Quay that security is extended with Clair which can scan their images as they are placed into the registry for known vulnerabilities.    The binary combo of Quay and Clair makes it a great combination.  However along with the sense of security comes the need for a smooth transition when upgrading Quay and Clair.   In the following blog I will demonstrate how I upgraded my Quay & Clair  3.3.4 environment to 3.5.4.   

The upgrade of Quay & Clair from 3.3.4 to 3.5.4 one would think is very trivial but there are a few things that change along the way which means the order of operation is important here.  Some of those changes are as follows and ones I experienced:

  • From 3.3.4 to 3.4.5 there is a database schema update that is performed so one will want to ensure they have a good backup of their MySQL or PostgreSQL.
  • From 3.3.4 to 3.4.5 the Clair scanning version changes from version 2 to version 4.  This means we first need to configure a brand new PostgreSQL database and then configure Clair and Quay for v4 scanning.
  • From 3.3.4 to 3.4.5 SSL certs now require to have Subject Alternative Names (SAN)

 Before we start the upgrade lets look at the current running environment.  In all my examples the docker command is used.  This could easily be replaced with podman and even in the official documentation podman is the preferred command from Quay 3.4+.

First lets look at the running containers by running the docker ps command:

# docker ps
CONTAINER ID     IMAGE                                             COMMAND                  CREATED        STATUS      PORTS                                                            NAMES
713cb9e63813     quay.io/redhat/clair-jwt:v3.3.4                   "/clair/clair-entr..."   3 hours ago    Up 3 hours  0.0.0.0:6060-6061->6060-6061/tcp                                 youthful_ptolemy
32c922276bf4     quay.io/redhat/quay:v3.3.4                        "/quay-registry/qu..."   3 hours ago    Up 3 hours  7443/tcp, 9091/tcp, 0.0.0.0:80->8080/tcp, 0.0.0.0:443->8443/tcp  blissful_thompson
1b05a8997caa     registry.access.redhat.com/rhscl/mysql-57-rhel7   "container-entrypo..."   3 hours ago    Up 3 hours  0.0.0.0:3306->3306/tcp                                           mysql
9de27908c351     postgres                                          "docker-entrypoint..."   3 hours ago    Up 3 hours  0.0.0.0:5432->5432/tcp                                           postgres
53b420f0c0be     registry.access.redhat.com/rhscl/redis-32-rhel7   "container-entrypo..."   3 hours ago    Up 3 hours  0.0.0.0:6379->6379/tcp                                           jovial_sinoussi

From the output above we can see that in this environment all processes run on the same server with the exception of the S3 storage that actually stores our registry data.   So lets break down what is running:

Container ID

Process

Purpose

External Port

Internal Port

713cb9e63813

Clair 3.3.4

Image Scanning

6060 & 6061

6060 & 6061

9de27908c351

PostgreSQL 

DB for Clair

5432

5432

32c922276bf4

Quay 3.3.4

Registry

80 & 443

8080 & 8443

1b05a8997caa

MySQL

DB for Quay

3306

3306

53b420f0c0be

Redis

DB Cache

6379

6379


Next I want to show the firewalld rules we have allowed.  Note that the list below also shows the additional ports I want opened for the upgrade process.

# firewall-cmd --list-all
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: eth0
  sources: 
  services: dhcpv6-client ssh
  ports: 8443/tcp 80/tcp 443/tcp 3306/tcp 6379/tcp 5432/tcp 6060/tcp 6161/tcp 6061/tcp 6062/tcp 6063/tcp 5433/tcp 8080/tcp 8081/tcp 8089/tcp
  protocols: 
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules:

Now lets take a look at the current Quay config.yaml file:

# cat /mnt/quay/config/config.yaml
AUTHENTICATION_TYPE: Database
BITTORRENT_FILENAME_PEPPER: af936296-6a1f-444a-8642-a2cd26184cb4
BUILDLOGS_REDIS:
  host: quay.schmaustech.com
  port: 6379
DATABASE_SECRET_KEY: '19413931088941115129071567106986409590936979178495908481080155841238934557252'
DB_URI: mysql+pymysql://quayuser:JzxCTamgFBmHRhcGFtoPHFkrx1BH2vwQ@192.168.0.11/enterpriseregistrydb
DEFAULT_TAG_EXPIRATION: 2w
DISTRIBUTED_STORAGE_CONFIG:
  default:
  - RadosGWStorage
  - access_key: BKIKJAA5BMMU2RHO6IBB
    bucket_name: schmaustech
    hostname: 192.168.0.21
    is_secure: false
    port: '9100'
    secret_key: V7f1CwQqAcwo80UEIJEjc5gVQUSSx5ohQ9GSrr12
    storage_path: /datastorage/registry
DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS: []
DISTRIBUTED_STORAGE_PREFERENCE:
- default
ENTERPRISE_LOGO_URL: /static/img/RH_Logo_Quay_Black_UX-horizontal.svg
FEATURE_ACI_CONVERSION: false
FEATURE_ANONYMOUS_ACCESS: true
FEATURE_APP_REGISTRY: false
FEATURE_APP_SPECIFIC_TOKENS: true
FEATURE_BUILD_SUPPORT: false
FEATURE_CHANGE_TAG_EXPIRATION: true
FEATURE_DIRECT_LOGIN: true
FEATURE_MAILING: false
FEATURE_PARTIAL_USER_AUTOCOMPLETE: true
FEATURE_REPO_MIRROR: false
FEATURE_REQUIRE_TEAM_INVITE: true
FEATURE_RESTRICTED_V1_PUSH: true
FEATURE_SECURITY_NOTIFICATIONS: true
FEATURE_SECURITY_SCANNER: true
FEATURE_USERNAME_CONFIRMATION: true
FEATURE_USER_CREATION: true
FEATURE_USER_LOG_ACCESS: true
GITHUB_LOGIN_CONFIG: {}
GITHUB_TRIGGER_CONFIG: {}
GITLAB_TRIGGER_KIND: {}
GPG2_PRIVATE_KEY_FILENAME: signing-private.gpg
GPG2_PUBLIC_KEY_FILENAME: signing-public.gpg
LOGS_MODEL: database
LOGS_MODEL_CONFIG: {}
LOG_ARCHIVE_LOCATION: default
MAIL_DEFAULT_SENDER: support@quay.io
MAIL_PORT: 587
MAIL_USE_TLS: true
PREFERRED_URL_SCHEME: https
REGISTRY_TITLE: Red Hat Quay
REGISTRY_TITLE_SHORT: Red Hat Quay
REPO_MIRROR_SERVER_HOSTNAME: null
REPO_MIRROR_TLS_VERIFY: true
SECRET_KEY: '61665332226336867411689064816174816874267671771605676993213345229382029684425'
SECURITY_SCANNER_ENDPOINT: http://quay.schmaustech.com:6060
SECURITY_SCANNER_ISSUER_NAME: security_scanner
SERVER_HOSTNAME: quay.schmaustech.com
SETUP_COMPLETE: true
SIGNING_ENGINE: gpg2
SUPER_USERS:
- admin
TAG_EXPIRATION_OPTIONS:
- 0s
- 1d
- 1w
- 2w
- 4w
TEAM_RESYNC_STALE_TIME: 60m
TESTING: false
USERFILES_LOCATION: default
USERFILES_PATH: userfiles/
USER_EVENTS_REDIS:
  host: quay.schmaustech.com
  port: 6379
USE_CDN: false

And then lets take a look at the current Clair config.yaml keeping in mind this is Clair v2 and during the upgrade process to Quay 3.4.3 we will be upgrading to Clair v4 so this file will be replaced:

# cat /mnt/clair/config/config.yaml 
clair:
  database:
    type: pgsql
    options:
      # A PostgreSQL Connection string pointing to the Clair Postgres database.
      # Documentation on the format can be found at: http://www.postgresql.org/docs/9.4/static/libpq-connect.html
      source: postgresql://postgres:password@quay.schmaustech.com:5432/clairtest?sslmode=disable 
      cachesize: 16384
  api:
    #The port at which Clair will report its health status. For example, if Clair is running at
    #https://clair.mycompany.com, the health will be reported at
    #http://clair.mycompany.com:6061/health.
    healthport: 6061

    port: 6062
    timeout: 900s

    # paginationkey can be any random set of characters. *Must be the same across all Clair instances*.
    paginationkey:

  updater:
    # interval defines how often Clair will check for updates from its upstream vulnerability databases.
    interval: 6h
  notifier:
    attempts: 3
    renotifyinterval: 1h
    http:
      # QUAY_ENDPOINT defines the endpoint at which Quay is running.
      # For example: https://myregistry.mycompany.com
      endpoint: https://quay.schmaustech.com/secscan/notify
      proxy: http://localhost:6063

jwtproxy:
  signer_proxy:
    enabled: true
    listen_addr: :6063
    ca_key_file: /certificates/mitm.key # Generated internally, do not change.
    ca_crt_file: /certificates/mitm.crt # Generated internally, do not change.
    signer:
      issuer: security_scanner
      expiration_time: 5m
      max_skew: 1m
      nonce_length: 32
      private_key:
        type: autogenerated
        options:
          rotate_every: 12h
          key_folder: /clair/config/
          key_server:
            type: keyregistry
            options:
              # QUAY_ENDPOINT defines the endpoint at which Quay is running.
              # For example: https://myregistry.mycompany.com
              registry: https://quay.schmaustech.com/keys/


  verifier_proxies:
  - enabled: true
    # The port at which Clair will listen.
    listen_addr: :6060

    # If Clair is to be served via TLS, uncomment these lines. See the "Running Clair under TLS"
    # section below for more information.
    #key_file: /clair/config/domain.key
    #crt_file: /clair/config/domain.crt

    verifier:
      # CLAIR_ENDPOINT is the endpoint at which this Clair will be accessible. Note that the port
      # specified here must match the listen_addr port a few lines above this.
      # Example: https://myclair.mycompany.com:6060
      audience: http://quay.schmaustech.com:6060

      upstream: http://localhost:6062
      key_server:
        type: keyregistry
        options:
          # QUAY_ENDPOINT defines the endpoint at which Quay is running.
          # Example: https://myregistry.mycompany.com
          registry: https://quay.schmaustech.com/keys/

At this point we have looked at the current running environment but before we begin upgrade I would recommend taking a Quay database backup whether its running MySQL or PostgreSQL.  We should do this because during the upgrade process of Quay the DB schema will get updated and so if we had to revert back we would need to restore from backup.

Now lets start the upgrade process from 3.3.4 to 3.4.5.   During this upgrade process we will be upgrading three components:  Redis, Clair & Quay.  Lets start with the Redis upgrade first as we can do this with the current Quay environment.   Below we will stop the current Redis container and then start the new Redis container for the 3.4.3 environment:

# docker stop 53b420f0c0be
53b420f0c0be

# docker run -d --rm --name redis -p 6379:6379 -e REDIS_PASSWORD=strongpassword registry.redhat.io/rhel8/redis-5:1
84f42084adae1e2e9d5b11aa8e06b90dde49dce015b8f30e72541631c8e98e05

# docker ps
CONTAINER ID        IMAGE                                       COMMAND                  CREATED             STATUS              PORTS                       NAMES
84f42084adae        registry.redhat.io/rhel8/redis-5:1          "container-entrypo..."   5 seconds ago       Up 4 seconds        0.0.0.0:6379->6379/tcp      redis

Since we added a password switch to Redis on the new version we need to also update the Quay config.yaml to include that password:

BUILDLOGS_REDIS:
    host: quay.schmaustech.com
    password: strongpassword
    port: 6379
    
USER_EVENTS_REDIS:
    host: quay.schmaustech.com
    password: strongpassword
    port: 6379

Now lets stop and start the Quay 3.3.4 container and confirm our Redis change takes effect:

# docker stop 32c922276bf4
32c922276bf4

# docker run --restart=always -p 443:8443 -p 80:8080    --sysctl net.core.somaxconn=4096    --privileged=true    -v /mnt/quay/config:/conf/stack:Z    -v /mnt/quay/storage:/datastorage:Z    -d quay.io/redhat/quay:v3.3.4
f0ab641a5d253532e263d38820ca1849f17428e0d7d12b6f79b676222d310b06

# docker ps
CONTAINER ID    IMAGE                        COMMAND                  CREATED              STATUS              PORTS                                                             NAMES
f0ab641a5d25    quay.io/redhat/quay:v3.3.4   "/quay-registry/qu..."   About a minute ago   Up About a minute   7443/tcp, 9091/tcp, 0.0.0.0:80->8080/tcp, 0.0.0.0:443->8443/tcp   gifted_einstein

Now lets move onto the upgrade of the Clair component.  For this we will need a new database for Clair to consume.    In my example I am going to run a new PostgreSQL database container completely separate from the current PostgreSQL container we originally had for our 3.3.4 environment.  To do this I will run the following commands:

# mkdir /mnt/postgres-clair4
# setfacl -m u:26:-wx /mnt/postgres-clair4
# docker run -d --rm --name postgresql-clair4 -e POSTGRESQL_USER=clairuser -e POSTGRESQL_PASSWORD=clairpass -e POSTGRESQL_DATABASE=clair -e POSTGRESQL_ADMIN_PASSWORD=password -p 5433:5432 -v /mnt/postgres-clair4:/var/lib/pgsql/data:Z -d registry.redhat.io/rhel8/postgresql-10:1
2b4b1afa3345d24bb6058c713e2f4d363a09da5bff75242d18bad69ddb720d34

# docker ps
CONTAINER ID        IMAGE                                           COMMAND                  CREATED             STATUS              PORTS                     NAMES
2b4b1afa3345        registry.redhat.io/rhel8/postgresql-10:1        "container-entrypo..."   3 seconds ago       Up 2 seconds        0.0.0.0:5433->5432/tcp    postgresql-clair4


Because Clair requires the uuid-ossp extensions in the PostgreSQL database we need to run the following SQL command against the database:

# docker exec -it postgresql-clair4 /bin/bash -c 'echo "CREATE EXTENSION IF NOT EXISTS \"uuid-ossp\"" | psql -d clair -U postgres'
CREATE EXTENSION

With the Clair v4 PostgreSQL database running lets turn our attention to the Clair v4 config.yaml.  Below I created my Clair config path and then added the contents to the config.yaml:

# mkdir -p /mnt/clair4/config
# cat /mnt/clair4/config/config.yaml 
http_listen_addr: :8081
introspection_addr: :8089
log_level: debug
indexer:
  connstring: host=quay.schmaustech.com port=5433 dbname=clair user=clairuser password=clairpass sslmode=disable
  scanlock_retry: 10
  layer_scan_concurrency: 5
  migrations: true
matcher:
  connstring: host=quay.schmaustech.com port=5433 dbname=clair user=clairuser password=clairpass sslmode=disable
  max_conn_pool: 100
  run: ""
  migrations: true
  indexer_addr: clair-indexer
notifier:
  connstring: host=quay.schmaustech.com port=5433 dbname=clair user=clairuser password=clairpass sslmode=disable
  delivery_interval: 1m
  poll_interval: 5m
  migrations: true
auth:
  psk:
    key: "MTU5YzA4Y2ZkNzJoMQ=="
    iss: ["quay"]
# tracing and metrics
trace:
  name: "jaeger"
  probability: 1
  jaeger:
    agent_endpoint: "localhost:6831"
    service_name: "clair"
metrics:
  name: "prometheus"

Before we start up the Clair v4 container though lets go ahead and make the changes we need to the Quay config.yaml as well.  Add or verify the following lines in the configuration:

FEATURE_SECURITY_NOTIFICATIONS: false
FEATURE_SECURITY_SCANNER: true
SECURITY_SCANNER_INDEXING_INTERVAL: 30
SECURITY_SCANNER_V4_ENDPOINT: http://quay.schmaustech.com:8081
SECURITY_SCANNER_V4_PSK: MTU5YzA4Y2ZkNzJoMQ==
SERVER_HOSTNAME: quay.schmaustech.com

At this point we are ready to stop the current Quay and Clair containers:

# docker stop 713cb9e63813 f0ab641a5d25 9de27908c351
713cb9e63813
f0ab641a5d25
9de27908c351

# docker rm 713cb9e63813 f0ab641a5d25 9de27908c351
713cb9e63813
f0ab641a5d25
9de27908c351

Next lets start up the Clair v4 container:

# docker run -d --rm --name clairv4 -p 8081:8081 -p 8089:8089 -e CLAIR_CONF=/clair/config.yaml -e CLAIR_MODE=combo -v /mnt/clair4/config:/clair:Z registry.redhat.io/quay/clair-rhel8:v3.4.5
b627ef1ee4669341d8a5fe72c095e14bb8931200a08d016f95faaec19d466423
[root@quay config]# docker ps
CONTAINER ID        IMAGE                                             COMMAND                  CREATED             STATUS              PORTS                                                      NAMES
b627ef1ee466        registry.redhat.io/quay/clair-rhel8:v3.4.5        "/usr/bin/dumb-ini..."   3 seconds ago       Up 1 second         0.0.0.0:8081->8081/tcp, 6060/tcp, 0.0.0.0:8089->8089/tcp   clairv4And finally lets start up the new Quay 3.4.3 container:

Once Clair v4 is up and running we can then start Quay up on version 3.4.5.  Note however that the image scanning may need some additional time to come up as Clair is repopulating the database of vulnerabilities.

# docker run -d --rm -p 443:8443 -p 8080:8080 --name quay -v /mnt/quay/config:/conf/stack:Z -v /mnt/quay/storage:/datastorage:Z registry.redhat.io/quay/quay-rhel8:v3.4.5
1f127bd3791e600a9bcf6f6e20f1fb1e368de27868da802f9ec8d6e5bf87310a
# docker ps
CONTAINER ID        IMAGE                                             COMMAND                  CREATED             STATUS              PORTS                                                      NAMES
1f127bd3791e        registry.redhat.io/quay/quay-rhel8:v3.4.5         "dumb-init -- /qua..."   4 seconds ago       Up 4 seconds        7443/tcp, 0.0.0.0:8080->8080/tcp, 0.0.0.0:443->8443/tcp    quay


If everything went well we should now have a Quay 3.4.3 and Clair 3.4.5 environment.   Before I proceed I want to validate that I can still push/pull content from my registry and that Clair v4 is scanning images appropriately.


Also this is a good time to quickly look at what the running environment should look like:

# docker ps
CONTAINER ID        IMAGE                                           COMMAND                  CREATED           STATUS            PORTS                                                      NAMES
1f127bd3791e        registry.redhat.io/quay/quay-rhel8:v3.4.5       "dumb-init -- /qua..."   35 minutes ago    Up 35 minutes     7443/tcp, 0.0.0.0:8080->8080/tcp, 0.0.0.0:443->8443/tcp  quay 
627ef1ee466        registry.redhat.io/quay/clair-rhel8:v3.4.5      "/usr/bin/dumb-ini..."   38 minutes ago     Up 38 minutes     0.0.0.0:8081->8081/tcp, 6060/tcp, 0.0.0.0:8089->8089/tcp  clairv4
2b4b1afa3345        registry.redhat.io/rhel8/postgresql-10:1        "container-entrypo..."   2 hours ago       Up 2 hours        0.0.0.0:5433->5432/tcp                                      postgresql-clair4
84f42084adae        registry.redhat.io/rhel8/redis-5:1              "container-entrypo..."   2 hours ago       Up 2 hours        0.0.0.0:6379->6379/tcp                                      redis
1b05a8997caa        registry.access.redhat.com/rhscl/mysql-57-rhel7 "container-entrypo..."   20 hours ago      Up 20 hours       0.0.0.0:3306->3306/tcp                                      mysql

Here is the chart view again of the environment breakdown:

Container ID

Process

Purpose

External Port

Internal Port

f8f4f441a89d

Clair 3.4.5

Image Scanning

8080 & 8081

8080 & 8081

2b4b1afa3345

PostgreSQL 

DB for Clair

5433

5432

1f127bd3791e

Quay 3.4.5

Registry

8080 & 443

8080 & 8443

1b05a8997caa

MySQL

DB for Quay

3306

3306

84f42084adae

Redis

DB Cache

6379

6379


At this point I am feeling confident we can move forward with the upgrade process to Quay 3.5.4.   Thankfully this step is not as complex as the move from 3.3.4 to 3.4.5.  In fact its just a matter of switching out the images being used for the containers.  The first step then is to stop the current Quay and Clair containers:

# docker stop f8f4f441a89d 1f127bd3791e
f8f4f441a89d
1f127bd3791e


In Quay 3.5.4 Helm & OCI artifacts have been enabled by default however since this is an upgrade we need to add those feature switches to the Quay config.yaml if we want them enabled:

FEATURE_GENERAL_OCI_SUPPORT: true
FEATURE_HELM_OCI_SUPPORT: true

Once we have stopped the current running containers for Quay and Clair and we have made any changes we needed to the config.yaml we can begin to start the Clair 3.5.4 container:

# docker run -d --rm --name clairv4 -p 8081:8081 -p 8089:8089 -e CLAIR_CONF=/clair/config.yaml -e CLAIR_MODE=combo -v /mnt/clair4/config:/clair:Z registry.redhat.io/quay/clair-rhel8:v3.5.4
0a046b625f4ef348e43a181db81341772cd690162df5eb34478dc525fabd591b
# docker ps
CONTAINER ID    IMAGE                                         COMMAND                CREATED         STATUS          PORTS                                                      NAMES
0a046b625f4e    registry.redhat.io/quay/clair-rhel8:v3.5.4    "/usr/bin/dumb-ini..." 3 seconds ago   Up 2 seconds    0.0.0.0:8081->8081/tcp, 6060/tcp, 0.0.0.0:8089->8089/tcp   clairv4

Once Clair is running we can now start the Quay 3.5.4 container:

# docker run -d --rm -p 443:8443 -p 8080:8080 --name quay -v /mnt/quay/config:/conf/stack:Z -v /mnt/quay/storage:/datastorage:Z registry.redhat.io/quay/quay-rhel8:v3.5.4
264ff68abb49fd53b0a2debeffd9684fb6504fa1558b874baadd271ac7a08225
# docker ps
CONTAINER ID   IMAGE                                       COMMAND                CREATED        STATUS         PORTS                                                      NAMES
264ff68abb49   registry.redhat.io/quay/quay-rhel8:v3.5.4   "dumb-init -- /qua..." 4 seconds ago  Up 3 seconds   7443/tcp, 0.0.0.0:8080->8080/tcp, 0.0.0.0:443->8443/tcp    quay

At this point we should again confirm that we can push/pull images from the Quay environment and also confirm that any new images are being scanned properly by Clair.



If everything checks out and works then we can declare success!