25 August 2023
Harbor in OKD
Previously we talked about how to use Harbor as image cache and how to configure CRI-O to use Harbor as cache. Today we want to show how to achieve the same behavior in an OKD cluster, whose configuration presents some additional difficulties.
Configuring OKD nodes
Supongamos que tenemos nuestro Harbor corriendo en my-harbor.example.com y que
ya tenemos configurado el proyecto de harbor para que funcione como cache de
docker.io. Además, OKD utiliza CRI-O, así que lo único
que deberíamos hacer es configurarlo para que utilice nuestro registry como
mirror, ¿verdad?
Suppose we have our Harbor instance running on my-harbor.example.com and that
we already have configured the Harbor project to work as cache to docker.io.
Since OKD uses CRI-O, we should just configure it to use
our registry as mirror, right?
Veamos, lo que queremos es reemplazar el archivo
/etc/containers/registries.conf en el filesystem de cada nodo. Según la
documentación oficial
de OKD, la configuración de las maquinas se realiza a través de un objeto
especial, los MachineConfigs. Estos se generan a partir de una definición
yaml:
Let’s see, what we really need to do is to replace the file
/etc/containers/registries.conf in each node’s filesystem. According to the
official OKD documentation,
machine configuration is done through a special object, the MachineConfigs.
These are generated from a yaml definition:
❯ cat 99-worker-harbor.bu
variant: openshift
version: 4.11.0
metadata:
name: 99-worker-harbor-proxy
labels:
machineconfiguration.openshift.io/role: worker
storage:
files:
- path: /etc/containers/registries.conf
mode: 0644
overwrite: true
contents:
inline: |
unqualified-search-registries = ['registry.access.redhat.com', 'docker.io']
[[registry]]
prefix = 'docker.io'
insecure = false
blocked = false
location = 'docker.io'
[[registry.mirror]]
location = 'my-harbor.example.com/proxy.docker.io'
insecure = true
Then the docs tells you to use an special tool, butane,
to generate the MachineConfigs manifests:
❯ butane 99-worker-harbor.bu -o ./99-worker-harbor.yaml
❯ cat 99-worker-harbor.yaml
# Generated by Butane; do not edit
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
labels:
machineconfiguration.openshift.io/role: worker
name: 99-worker-harbor-proxy
spec:
config:
ignition:
version: 3.2.0
storage:
files:
- contents:
compression: gzip
source: data:;base64,H4sIAAAAAAAC/2yOQW6EMAxF9zlFdtkU9wScBGVhEgNWQ0ztRILbV5VK2xnN8uk9W7/Xz46FF6Y8GKGmbVBa2ZoymR/9FH7wAkyJzEApb9ggyR7efMiSPkiBJUQ3TXcbozuUFj79+D9xXI1SV/KjX7AYubl8y/zLRRI2lvp09/cZdlYVjfEhve2woc6iYFTREFaZAfX9UDkveD2jaSf3FQAA//95sGm0BQEAAA==
mode: 420
overwrite: true
path: /etc/containers/registries.conf
But after applying it, although the machineconfigpools start to update, they
do not progress correctly and this error is shown:
pool is degraded because nodes fail with "1 nodes are reporting degraded
status on sync": "Node okd-cluster-j92p5-blackbox-monitoring-pmbmh is
reporting: \"failed decoding TOML content from file
/etc/containers/registries.conf: files cannot contain NULL bytes; probably
using UTF-16; TOML files must be UTF-8\""
We tried different ways to encode the file, but always got the same error.
Solution
La solución que encontramos fue encodear
el archivo en base64 y pegarlo en el campo source del manifiesto antes
generado. Para esto primero se escribe el archivo registries.conf con el
contenido deseado, por ejemplo:
The solution we found was to base64-encode
the file and paste it into the source field of the manifest generated above.
In order to do this, we first wrote a registries.conf file with the desired content, for example:
❯ cat registries.conf
unqualified-search-registries = ['registry.access.redhat.com','docker.io']
[[registry]]
prefix = 'docker.io'
insecure = false
blocked = false
location = 'docker.io'
[[registry.mirror]]
location = 'my-harbor.example.com/proxy.docker.io'
insecure = true
It is then encoded in base64:
❯ cat registries.conf | base64 -w0
dW5xdWFsaWZpZWQtc2VhcmNoLXJlZ2lzdHJpZXMgPSBbJ3JlZ2lzdHJ5LmFjY2Vzcy5yZWRoYXQuY29tJywnZG9ja2VyLmlvJ10KW1tyZWdpc3RyeV1dCnByZWZpeCA9ICdkb2NrZXIuaW8nCmluc2VjdXJlID0gZmFsc2UKYmxvY2tlZCA9IGZhbHNlCmxvY2F0aW9uID0gJ2RvY2tlci5pbycKW1tyZWdpc3RyeS5taXJyb3JdXQpsb2NhdGlvbiA9ICdteS1oYXJib3IuZXhhbXBsZS5jb20vcHJveHkuZG9ja2VyLmlvJwppbnNlY3VyZSA9IHRydWUK%
This is pasted into the source field of the previously generated file
(ignoring the “do not edit” comment). We also removed the gzip compression and
specified the charset with data:text/plain;charset=utf-8;base64,:
❯ cat 99-worker-harbor.yaml
# Generated by Butane; do not edit
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
labels:
machineconfiguration.openshift.io/role: worker
name: 99-worker-harbor-proxy
spec:
config:
ignition:
version: 3.2.0
storage:
files:
- contents:
source: data:text/plain;charset=utf-8;base64,dW5xdWFsaWZpZWQtc2VhcmNoLXJlZ2lzdHJpZXMgPSBbJ3JlZ2lzdHJ5LmFjY2Vzcy5yZWRoYXQuY29tJywnZG9ja2VyLmlvJ10KW1tyZWdpc3RyeV1dCnByZWZpeCA9ICdkb2NrZXIuaW8nCmluc2VjdXJlID0gZmFsc2UKYmxvY2tlZCA9IGZhbHNlCmxvY2F0aW9uID0gJ2RvY2tlci5pbycKW1tyZWdpc3RyeS5taXJyb3JdXQpsb2NhdGlvbiA9ICdteS1oYXJib3IuZXhhbXBsZS5jb20vcHJveHkuZG9ja2VyLmlvJwppbnNlY3VyZSA9IHRydWUK
mode: 420
overwrite: true
path: /etc/containers/registries.conf
Aplicando este manifiesto se empezarán a actualizar los nodos. Al finalizar se
se puede ingresar a alguno de ellos y realizar un pull de cualquier imagen de
docker.io con usando crictl pull. Si dicha imagen aparece en el proyecto de Harbor significa que la configuración se ha realizado exitosamente.
After applying this manifest the machine controller will begin updating nodes.
When finished you can ssh into any of them and pull some docker.io image
using crictl pull. If that image arrives into the Harbor project, it means
the configuration has been carried out successfully.
