DevOps
Sealed Secrets – secure way to manage secrets
Sealed Secrets provide a secure way to encrypt the Kubernetes secrets.
Sealed Secrets provide a secure way to encrypt the Kubernetes secrets. This resource can be stored in public repository and used in devops automation maintaining confidentiality of the secured information. We implement the sealed secret controller in the Kubernetes cluster to manage SealedSecret. SealedSecret can be decrypted to secret object in the target namespace and used as per Kubernetes behaviour.
Sealed Secrets is an open source project maintained by bitnami-labs.
Setup Sealed Secrets
To configure Sealed Secrets for use, we need to setup in following way :
CLI for client VM
The kubeseal client is available on homebrew: $ brew install kubeseal
The kubeseal utility uses asymmetric crypto to encrypt secrets that only the controller can decrypt. These encrypted secrets are encoded in a SealedSecret resource, which you can see as a recipe for creating a secret. It is expected to install Sealed Secret Controller with name as “sealed-secrets-controller” in Kube-system namespace by default. If we install the controller with different name then provide the name with --controller-name and for different namespace, need to provide the --controller-namespace in the kubeseal command.
An example for SealedSecret usage :
# Create secret resource$ kubectl create secret generic mysecret --from-literal=password=pass123 --dry-run=client -o yaml > mysecret.yaml# Convert secret to SealedSecret resource$ kubeseal -f mysecret.yaml -w mysecret-sealed.yaml$ cat mysecret-sealed.yamlapiVersion: bitnami.com/v1alpha1kind: SealedSecretmetadata: creationTimestamp: null name: mysecret namespace: defaultspec: encryptedData: password: AgCwKX17QKONR59+T9t04x+05QBYuFcIprAGTxbYXax7oZGT1Stc/qEPvhJO1wcxjppY2q5JIpcG11wdGSrzW6++DM5eaYti0F/HVH/+cjM3Xz2f75ZPYNml8zmXHvnHPqtmE3D7695MngtGJ4b40ayl/Z7hLwSInBZ0j7k+WrONVubb0+/s3AUQ96VqdUnVkYFWv7FVSK90JWrLX1OfoJtUdJYzoxsYkgDMOLLkpXAM4pj+S8ksAw7kw77qjErPL94qioHodKpP1QRohTsapNywBq5AfyDMbe9bD6u4UY7n9prP6SDmt99ZtMtJ0JuujPTMOlf7e1BAEZMxujqFZDPpulkoL0Hytbv3pX/827J93R+uyAT8zRS9HEGNPMDQUYlDj6LD3lkMIg6AceL58mlAzhCa+LEzGNKAV+9u7JWcdOGl9vulYJPqJBMmi1yCfEAEfSdq3qD1ZSjP5TsZAQqag12ZTr//QF5afCN8jyogvbZWyPV/BaqzrmGKgp2xYdrhdyCHYagMdU+OkhUucNXZqEDr+5JDJl++8S02cwugdksy7aVemWuWCBiYcP88N8UBIsHDt/M5ngko9qNvnoIorK5DKcsPUE00wK/RLPWkxmeSGupUGhpW9Bm67RLd/2jNan8aVBvuszHzMs9RYM9/dofHqHburaUKLa6qs0rWXTkRmESQDqq5Jc/zF0R9O/mWO7+LNiKG template: data: null metadata: creationTimestamp: null name: mysecret namespace: defaultkubeseal uses the cert as configured with the controller during the encryption process. If we are using kubeseal where cluster is unreachable, then we can pull the cert with kubeseal --fetch-cert command and then pass the certificate with --cert option to create the sealedSecret resource.
The above created “mysecret-sealed.yaml” can be safely shared across networks and in public/private archives and repo. Since it’s encrypted and require the private key ( placed only with controller) thus can’t be decoded externally. When we create the SealedSecret resource in the cluster, the controller recognises the resource and create the normal unencrypted secret in the namespace using the private/public key pairs maintained as kubernetes secrets at controller.
Controller for cluster
The Sealed Secrets helm chart is now official supported and hosted in this GitHub repo.
helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secretshelm install sealed-secrets-controller --namespace kube-system sealed-secrets/sealed-secretsThe controller generates its own self-signed certificates when is deployed for the first time and manages the renewal lifecycle of the certificates.
$ k describe secret sealed-secrets-keyrgcbw -n kube-systemName: sealed-secrets-keyrgcbwNamespace: kube-systemLabels: sealedsecrets.bitnami.com/sealed-secrets-key=activeAnnotations: <none>Type: kubernetes.io/tlsData====tls.crt: 1724 bytestls.key: 3247 bytesWe can provide own certificates, which can be created in the kube-system namespace (or same namespace wherever controller is deployed) with secret type as “kubernetes.io/tls” labeled with sealedsecrets.bitnami.com/sealed-secrets-key=active.
The sealed-secrets-controller can be further configured with various options to add or change features to the controller.
Usage of controller: --accept-deprecated-v1-data Accept deprecated V1 data field. (default true) --all-namespaces Scan all namespaces or only the current namespace (default=true). (default true) --key-cutoff-time string Create a new key if latest one is older than this cutoff time. RFC1123 format with numeric timezone expected. --key-prefix string Prefix used to name keys. (default "sealed-secrets-key") --key-renew-period duration New key generation period (automatic rotation disabled if 0) (default 720h0m0s) --key-size int Size of encryption key. (default 4096) --key-ttl duration Duration that certificate is valid for. (default 87600h0m0s) --label-selector string Label selector which can be used to filter sealed secrets. --listen-addr string HTTP serving address. (default ":8080") --my-cn string Common name to be used as issuer/subject DN in generated certificate. --old-gc-behaviour Revert to old GC behavior where the controller deletes secrets instead of delegating that to k8s itself. --read-timeout duration HTTP request timeout. (default 2m0s) --update-status beta: if true, the controller will update the status subresource whenever it processes a sealed secret (default true) --version Print version information and exit --write-timeout duration HTTP response timeout. (default 2m0s)Sealed key renewal
Auto configuration
Sealed key renewal is configured to 30d by default. This value can be changed with the controller option “--key-renew-period”.
Manual process
There might be a need to rotate the private/public key pair attached to controller earlier then it’s expiry period. This might be need for any suspicion for compromised key or any other use case. Theprivate/public key pair can be generated early by passing current timestamp to the controller into a flag --key-cutoff-time or with an environment variable called SEALED_SECRETS_KEY_CUTOFF_TIME. Expected date format is RFC1123, which can be generated with the date -R command.
$ date -RFri, 17 Dec 2021 23:35:48 +0800$ helm upgrade sealed-secrets-controller --namespace kube-system sealed-secrets/sealed-secrets --set commandArgs[0]=--key-cutoff-time\='Fri\, 17 Dec 2021 23:35:48 +0800'With the above helm upgrade, the new secret would be created with a set ofprivate/public key pairs and get attached to the controller. It would be applicable for any new SealedSecret operation within the cluster.
Backup and recovery
To prepare for any cluster operation tasks, we can create a backup of the secret attached to the controller which holds the cert and the key. The key is used for decrypt operation for sealed secrets into secrets.
kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml > backup-sealedSecret.key
In order to attach this secret to the controller, we can create the secret in the namespace and delete the controller pod to let the pod get recreated and cache the provisioned secret with the backup keys.
Also, there might be a need ( usually it’s not recommended) to recover the secret from the sealed secret privately in secured environment for review or any other legitimate use case. For that, we can use the backed up keys from the controller to use during the recovery process.
$ kubeseal --recovery-unseal -f mysecret-sealed.yaml --recovery-private-key backup-sealedSecret.key -o yamlapiVersion: v1data: password: cGFzczEyMw==kind: Secretmetadata: creationTimestamp: null name: mysecret namespace: default ownerReferences: - apiVersion: bitnami.com/v1alpha1 controller: true kind: SealedSecret name: mysecret uid: ""Usage of SealedSecret
In the below example, we would deploy MYSQL DB application in the Kubernetes Cluster. The DB needs to be configured with the root password, passed by the environment variable – MYSQL_ROOT_PASSWORD to the deployment resource. We would create a sealedSecret resource for the root password, which would generate the secret resource in the defined namespace. Once MYSQLDB would be deployed, it would refer to the secret resource and fetch the root password to configure the DB application.
Create Namespace for the app deployment
$ kubectl create ns database-nsnamespace/database-ns createdCreate secret for root password and push to kubeseal tool to create sealedSecret
kubectl create secret generic mysqlpass --from-literal=password=Password123 -n database-ns --dry-run=client -o yaml | kubeseal -o yaml > mysqlpass-sealedsecret.yamlVerify the sealedSecret, to see that the name and namespace match to the secret which follows the default “strict” scope. We can refer here, for further details on other scopes – namespace-wide and cluster-wide.
$ cat mysqlpass-sealedsecret.yamlapiVersion: bitnami.com/v1alpha1kind: SealedSecretmetadata: creationTimestamp: null name: mysqlpass namespace: database-nsspec: encryptedData: password: <<Encrypted-data>> template: data: null metadata: creationTimestamp: null name: mysqlpass namespace: database-nsCreate SealedSecret resource in cluster and verify that the sealed secret and secret both gets created in the database-ns.
$ kubectl create -f mysqlpass-sealedsecret.yamlsealedsecret.bitnami.com/mysqlpass created$ kubectl get secret,sealedsecret -n database-nsNAME TYPE DATA AGEsecret/default-token-hbj8k kubernetes.io/service-account-token 3 4m37ssecret/mysqlpass Opaque 1 43sNAME AGEsealedsecret.bitnami.com/mysqlpass 43sCreate the MYSQL DB in the database-ns namespace.
$ cat mysqldb.yamlapiVersion: apps/v1kind: Deploymentmetadata: name: mysql-server namespace: database-ns labels: app: mysqldbspec: selector: matchLabels: app: mysqldb tier: db strategy: type: Recreate template: metadata: labels: app: mysqldb tier: db spec: containers: - image: mysql:5.6 name: mysql env: - name: MYSQL_ROOT_PASSWORD valueFrom: secretKeyRef: name: mysqlpass key: password ports: - containerPort: 3306 name: mysql$ kubectl create -f mysqldb.yamldeployment.apps/mysql-server createdVerify that the mysql db is configured and able to launch with the defined root password.
$ kubectl get pods -n database-nsNAME READY STATUS RESTARTS AGEpod/mysql-server-666bb9ffdb-dclcq 1/1 Running 0 49s$ kubectl exec -it -n database-ns mysql-server-666bb9ffdb-dclcq -- /bin/bashroot@mysql-server-666bb9ffdb-dclcq:/#root@mysql-server-666bb9ffdb-dclcq:/# mysql -u root -pEnter password:Welcome to the MySQL monitor. Commands end with ; or \g.Your MySQL connection id is 2Server version: 5.6.51 MySQL Community Server (GPL)Copyright (c) 2000, 2021, Oracle and/or its affiliates. All rights reserved.Oracle is a registered trademark of Oracle Corporation and/or itsaffiliates. Other names may be trademarks of their respectiveowners.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.mysql> show databases;+--------------------+| Database |+--------------------+| information_schema || mysql || performance_schema |+--------------------+3 rows in set (0.00 sec)Wrapping Up
With the above implementation, we can conclude that SealedSecret adds an encryption to secure data for archival purpose, and can be used to deploy secrets in cluster. For the applications in cluster, it still uses the secret object generated by the underlying sealed secrets.
Thus, with this innovative approach, we get more secure way to store and use secrets for public repository and devops automation purpose.