Sealed Secrets – secure way to manage secrets

Sealed Secrets provide a secure way to encrypt the Kubernetes secrets. This resource can be stored in public repository and used in devops automation maintaining confidentiality of the secured information. We implement the sealed secret controller in the Kubernetes cluster to manage SealedSecret. SealedSecret can be decrypted to secret object in the target namespace and used as per Kubernetes behaviour.

Sealed Secrets is an open source project maintained by bitnami-labs.

Setup Sealed Secrets

To configure Sealed Secrets for use, we need to setup in following way :

CLI for client VM

The kubeseal client is available on homebrew: $ brew install kubeseal

The kubeseal utility uses asymmetric crypto to encrypt secrets that only the controller can decrypt. These encrypted secrets are encoded in a SealedSecret resource, which you can see as a recipe for creating a secret. It is expected to install Sealed Secret Controller with name as "sealed-secrets-controller" in Kube-system namespace by default. If we install the controller with different name then provide the name with --controller-name and for different namespace, need to provide the --controller-namespace in the kubeseal command.

An example for SealedSecret usage :

 
# Create secret resource
$ kubectl create secret generic mysecret --from-literal=password=pass123 --dry-run=client -o yaml > mysecret.yaml
 
# Convert secret to SealedSecret resource
$ kubeseal -f mysecret.yaml -w mysecret-sealed.yaml
 
$ cat mysecret-sealed.yaml
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  creationTimestamp: null
  name: mysecret
  namespace: default
spec:
  encryptedData:
    password: AgCwKX17QKONR59+T9t04x+05QBYuFcIprAGTxbYXax7oZGT1Stc/qEPvhJO1wcxjppY2q5JIpcG11wdGSrzW6++DM5eaYti0F/HVH/+cjM3Xz2f75ZPYNml8zmXHvnHPqtmE3D7695MngtGJ4b40ayl/Z7hLwSInBZ0j7k+WrONVubb0+/s3AUQ96VqdUnVkYFWv7FVSK90JWrLX1OfoJtUdJYzoxsYkgDMOLLkpXAM4pj+S8ksAw7kw77qjErPL94qioHodKpP1QRohTsapNywBq5AfyDMbe9bD6u4UY7n9prP6SDmt99ZtMtJ0JuujPTMOlf7e1BAEZMxujqFZDPpulkoL0Hytbv3pX/827J93R+uyAT8zRS9HEGNPMDQUYlDj6LD3lkMIg6AceL58mlAzhCa+LEzGNKAV+9u7JWcdOGl9vulYJPqJBMmi1yCfEAEfSdq3qD1ZSjP5TsZAQqag12ZTr//QF5afCN8jyogvbZWyPV/BaqzrmGKgp2xYdrhdyCHYagMdU+OkhUucNXZqEDr+5JDJl++8S02cwugdksy7aVemWuWCBiYcP88N8UBIsHDt/M5ngko9qNvnoIorK5DKcsPUE00wK/RLPWkxmeSGupUGhpW9Bm67RLd/2jNan8aVBvuszHzMs9RYM9/dofHqHburaUKLa6qs0rWXTkRmESQDqq5Jc/zF0R9O/mWO7+LNiKG
  template:
    data: null
    metadata:
      creationTimestamp: null
      name: mysecret
      namespace: default

kubeseal uses the cert as configured with the controller during the encryption process. If we are using kubeseal where cluster is unreachable, then we can pull the cert with kubeseal --fetch-cert command and then pass the certificate with --cert option to create the sealedSecret resource.

The above created “mysecret-sealed.yaml” can be safely shared across networks and in public/private archives and repo. Since it’s encrypted and require the private key ( placed only with controller) thus can’t be decoded externally. When we create the SealedSecret resource in the cluster, the controller recognises the resource and create the normal unencrypted secret in the namespace using the private/public key pairs maintained as kubernetes secrets at controller.

Controller for cluster

The Sealed Secrets helm chart is now official supported and hosted in this GitHub repo.

helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
 
helm install sealed-secrets-controller --namespace kube-system sealed-secrets/sealed-secrets

The controller generates its own self-signed certificates when is deployed for the first time and manages the renewal lifecycle of the certificates.

 
$ k describe secret sealed-secrets-keyrgcbw -n kube-system
Name:         sealed-secrets-keyrgcbw
Namespace:    kube-system
Labels:       sealedsecrets.bitnami.com/sealed-secrets-key=active
Annotations:  <none>
 
Type:  kubernetes.io/tls
 
Data
====
tls.crt:  1724 bytes
tls.key:  3247 bytes

Error

There is a minor bug in controller service for the helm version 1.17.1 To workaround, we need to remove the “name: http” for the Kubernetes service created to expose the controller.

We can provide own certificates, which can be created in the kube-system namespace (or same namespace wherever controller is deployed) with secret type as “kubernetes.io/tls” labeled with sealedsecrets.bitnami.com/sealed-secrets-key=active.

The sealed-secrets-controller can be further configured with various options to add or change features to the controller.

 
Usage of controller:
      --accept-deprecated-v1-data   Accept deprecated V1 data field. (default true)
      --all-namespaces              Scan all namespaces or only the current namespace (default=true). (default true)
      --key-cutoff-time string      Create a new key if latest one is older than this cutoff time. RFC1123 format with numeric timezone expected.
      --key-prefix string           Prefix used to name keys. (default "sealed-secrets-key")
      --key-renew-period duration   New key generation period (automatic rotation disabled if 0) (default 720h0m0s)
      --key-size int                Size of encryption key. (default 4096)
      --key-ttl duration            Duration that certificate is valid for. (default 87600h0m0s)
      --label-selector string       Label selector which can be used to filter sealed secrets.
      --listen-addr string          HTTP serving address. (default ":8080")
      --my-cn string                Common name to be used as issuer/subject DN in generated certificate.
      --old-gc-behaviour            Revert to old GC behavior where the controller deletes secrets instead of delegating that to k8s itself.
      --read-timeout duration       HTTP request timeout. (default 2m0s)
      --update-status               beta: if true, the controller will update the status subresource whenever it processes a sealed secret (default true)
      --version                     Print version information and exit
      --write-timeout duration      HTTP response timeout. (default 2m0s)

Sealed key renewal

Auto configuration

Sealed key renewal is configured to 30d by default. This value can be changed with the controller option "--key-renew-period".

Manual process

There might be a need to rotate the private/public key pair attached to controller earlier then it’s expiry period. This might be need for any suspicion for compromised key or any other use case. Theprivate/public key pair can be generated early by passing current timestamp to the controller into a flag --key-cutoff-time or with an environment variable called SEALED_SECRETS_KEY_CUTOFF_TIME. Expected date format is RFC1123, which can be generated with the date -R command.

 
$ date -R
Fri, 17 Dec 2021 23:35:48 +0800
 
$ helm upgrade sealed-secrets-controller --namespace kube-system sealed-secrets/sealed-secrets --set commandArgs[0]=--key-cutoff-time\='Fri\, 17 Dec 2021 23:35:48 +0800'

With the above helm upgrade, the new secret would be created with a set ofprivate/public key pairs and get attached to the controller. It would be applicable for any new SealedSecret operation within the cluster.

NOTE

If a sealing key has been leaked out of the cluster, we must consider all our SealedSecret resources encrypted with that key and stored in public repository as compromised. No amount of sealing key rotation in the cluster or even re-encryption of existing SealedSecrets files can change that. The best practice would be to periodically rotate all the actual secrets and create new SealedSecret resource with those new secrets. Also renew the sealed secret key if suspect of any compromise.

Backup and recovery

To prepare for any cluster operation tasks, we can create a backup of the secret attached to the controller which holds the cert and the key. The key is used for decrypt operation for sealed secrets into secrets.

kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml > backup-sealedSecret.key

In order to attach this secret to the controller, we can create the secret in the namespace and delete the controller pod to let the pod get recreated and cache the provisioned secret with the backup keys.

Also, there might be a need ( usually it’s not recommended) to recover the secret from the sealed secret privately in secured environment for review or any other legitimate use case. For that, we can use the backed up keys from the controller to use during the recovery process.

 
$ kubeseal --recovery-unseal -f mysecret-sealed.yaml --recovery-private-key backup-sealedSecret.key -o yaml
apiVersion: v1
data:
  password: cGFzczEyMw==
kind: Secret
metadata:
  creationTimestamp: null
  name: mysecret
  namespace: default
  ownerReferences:
  - apiVersion: bitnami.com/v1alpha1
    controller: true
    kind: SealedSecret
    name: mysecret
    uid: ""

Usage of SealedSecret

In the below example, we would deploy MYSQL DB application in the Kubernetes Cluster. The DB needs to be configured with the root password, passed by the environment variable – MYSQL_ROOT_PASSWORD to the deployment resource. We would create a sealedSecret resource for the root password, which would generate the secret resource in the defined namespace. Once MYSQLDB would be deployed, it would refer to the secret resource and fetch the root password to configure the DB application.

Create Namespace for the app deployment

$ kubectl create ns database-ns
namespace/database-ns created

Create secret for root password and push to kubeseal tool to create sealedSecret

 
kubectl create secret generic mysqlpass --from-literal=password=Password123 -n database-ns --dry-run=client -o yaml | kubeseal -o yaml > mysqlpass-sealedsecret.yaml

Verify the sealedSecret, to see that the name and namespace match to the secret which follows the default “strict” scope. We can refer here, for further details on other scopes – namespace-wide and cluster-wide.

 
$ cat mysqlpass-sealedsecret.yaml
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  creationTimestamp: null
  name: mysqlpass
  namespace: database-ns
spec:
  encryptedData:
    password: <<Encrypted-data>>
  template:
    data: null
    metadata:
      creationTimestamp: null
      name: mysqlpass
      namespace: database-ns

Create SealedSecret resource in cluster and verify that the sealed secret and secret both gets created in the database-ns.

 
$ kubectl create -f mysqlpass-sealedsecret.yaml
sealedsecret.bitnami.com/mysqlpass created
 
$ kubectl get secret,sealedsecret -n database-ns
NAME                         TYPE                                  DATA   AGE
secret/default-token-hbj8k   kubernetes.io/service-account-token   3      4m37s
secret/mysqlpass             Opaque                                1      43s
 
NAME                                 AGE
sealedsecret.bitnami.com/mysqlpass   43s

Create the MYSQL DB in the database-ns namespace.

 
$ cat mysqldb.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql-server
  namespace: database-ns
  labels:
     app: mysqldb
spec:
  selector:
    matchLabels:
      app: mysqldb
      tier: db
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: mysqldb
        tier: db
    spec:
      containers:
        - image: mysql:5.6
          name: mysql
          env:
          - name: MYSQL_ROOT_PASSWORD
            valueFrom:
              secretKeyRef:
                name: mysqlpass
                key: password
          ports:
          - containerPort: 3306
          name: mysql
 
$ kubectl create -f mysqldb.yaml
deployment.apps/mysql-server created

Verify that the mysql db is configured and able to launch with the defined root password.

 
$ kubectl get pods -n database-ns
NAME                                READY   STATUS    RESTARTS   AGE
pod/mysql-server-666bb9ffdb-dclcq   1/1     Running   0          49s
 
$ kubectl exec -it -n database-ns mysql-server-666bb9ffdb-dclcq -- /bin/bash
root@mysql-server-666bb9ffdb-dclcq:/#
root@mysql-server-666bb9ffdb-dclcq:/# mysql -u root -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.6.51 MySQL Community Server (GPL)
 
Copyright (c) 2000, 2021, Oracle and/or its affiliates. All rights reserved.
 
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
 
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
 
mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
+--------------------+
3 rows in set (0.00 sec)

Wrapping Up

With the above implementation, we can conclude that SealedSecret adds an encryption to secure data for archival purpose, and can be used to deploy secrets in cluster. For the applications in cluster, it still uses the secret object generated by the underlying sealed secrets.

Thus, with this innovative approach, we get more secure way to store and use secrets for public repository and devops automation purpose.