How to renew certificate manually in Kubernetes

Certificate renewal for client certificates becomes usual process for a long running Kubernetes clusters. This blog is an implementation of the process to perform manual cert upgrade. There might be multiple business reasons to perform manual cert rotation due to stringent business requirements. For the Kubernetes clusters installed with kubeadm, all the client certificates generated by kubeadm expire after 1 year. As a standard procedure, manual renewal is not usually required as cert rotation gets performed automatically when we perform Kubeadm upgrade.

For scenarios to perform manual certificate renewal, during the last month or week of expiry period, we can use Kubeadm commands to verify the certs and renew them for the cluster. The operations team can connect to control plane node and access the kubeadm utility to manage the certificates.

Kubeadm Cert management

Kubeadm is equipped with cert management and renewal commands.

[root@kmaster ~]# kubeadm certsCommands related to handling kubernetes certificatesUsage:  kubeadm certs [command]Available Commands:  certificate-key  Generate certificate keys  check-expiration Check certificates expiration for a Kubernetes cluster  generate-csr     Generate keys and certificate signing requests  renew            Renew certificates for a Kubernetes cluster

Kubeadm command for renew

Usage:  kubeadm certs renew [flags]  kubeadm certs renew [command]Available Commands:  admin.conf               Renew the certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself  all                      Renew all available certificates  apiserver                Renew the certificate for serving the Kubernetes API  apiserver-etcd-client    Renew the certificate the apiserver uses to access etcd  apiserver-kubelet-client Renew the certificate for the API server to connect to kubelet  controller-manager.conf  Renew the certificate embedded in the kubeconfig file for the controller manager to use  etcd-healthcheck-client  Renew the certificate for liveness probes to healthcheck etcd  etcd-peer                Renew the certificate for etcd nodes to communicate with each other  etcd-server              Renew the certificate for serving etcd  front-proxy-client       Renew the certificate for the front proxy client  scheduler.conf           Renew the certificate embedded in the kubeconfig file for the scheduler manager to use

Config and certs backup

For precautionary measure, it’s advised to perform backup for Kubernetes configs and certificates.

# Backup certsmkdir -p $HOME/k8scluster-old-certs/pki/bin/cp -p /etc/kubernetes/pki/*.* $HOME/k8scluster-old-certs/pkils -l $HOME/k8scluster-old-certs/pki/#Backup configs/bin/cp -p /etc/kubernetes/*.conf $HOME/k8scluster-old-certsls -ltr $HOME/k8scluster-old-certs#Backup local configmkdir -p $HOME/k8scluster-old-certs/.kube/bin/cp -p ~/.kube/config $HOME/k8scluster-old-certs/.kube/.ls -l $HOME/k8scluster-old-certs/.kube/.

Certificate Expiry Check

We can check the cert expiry period with follow command :

[root@kmaster ~]# kubeadm certs check-expiration[check-expiration] Reading configuration from the cluster...CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGEDadmin.conf                 Nov 19, 2022 14:53 UTC   344d                                    noapiserver                  Nov 19, 2022 14:53 UTC   344d            ca                      noapiserver-etcd-client      Nov 19, 2022 14:53 UTC   344d            etcd-ca                 noapiserver-kubelet-client   Nov 19, 2022 14:53 UTC   344d            ca                      nocontroller-manager.conf    Nov 19, 2022 14:53 UTC   344d                                    noetcd-healthcheck-client    Nov 19, 2022 14:53 UTC   344d            etcd-ca                 noetcd-peer                  Nov 19, 2022 14:53 UTC   344d            etcd-ca                 noetcd-server                Nov 19, 2022 14:53 UTC   344d            etcd-ca                 nofront-proxy-client         Nov 19, 2022 14:53 UTC   344d            front-proxy-ca          noscheduler.conf             Nov 19, 2022 14:53 UTC   344d                                    noCERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGEDca                      Nov 17, 2031 14:53 UTC   9y              noetcd-ca                 Nov 17, 2031 14:53 UTC   9y              nofront-proxy-ca          Nov 17, 2031 14:53 UTC   9y              no

The above command shows the expiration/residual time for the client certificates. These certificates are located at /etc/kubernetes/pki on the control plane nodes of Kubernetes cluster.

Implementation of manual certificate renewal

We can renew the certificates manually at any time with the kubeadm certs renew command. This command performs the renewal using CA certificate and key stored in /etc/kubernetes/pki.

For an HA Kubernetes cluster, kubeadm certs renew command needs to be executed on all the control-plane nodes.

Renew single certificate

[root@kmaster kubernetes]# kubeadm certs renew admin.conf[renew] Reading configuration from the cluster...certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed

Result of above step

[root@kmaster kubernetes]# kubeadm certs check-expiration[check-expiration] Reading configuration from the cluster...CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGEDadmin.conf                 Dec 10, 2022 05:53 UTC   364d                                    no.......

Renew all certificates

[root@kmaster kubernetes]# kubeadm certs renew all[renew] Reading configuration from the cluster...certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewedcertificate for serving the Kubernetes API renewedcertificate the apiserver uses to access etcd renewedcertificate for the API server to connect to kubelet renewedcertificate embedded in the kubeconfig file for the controller manager to use renewedcertificate for liveness probes to healthcheck etcd renewedcertificate for etcd nodes to communicate with each other renewedcertificate for serving etcd renewedcertificate for the front proxy client renewedcertificate embedded in the kubeconfig file for the scheduler manager to use renewedDone renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.

Result of above step

[root@kmaster kubernetes]# kubeadm certs check-expiration[check-expiration] Reading configuration from the cluster...CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGEDadmin.conf                 Dec 10, 2022 05:56 UTC   364d                                    noapiserver                  Dec 10, 2022 05:56 UTC   364d            ca                      noapiserver-etcd-client      Dec 10, 2022 05:56 UTC   364d            etcd-ca                 noapiserver-kubelet-client   Dec 10, 2022 05:56 UTC   364d            ca                      nocontroller-manager.conf    Dec 10, 2022 05:56 UTC   364d                                    noetcd-healthcheck-client    Dec 10, 2022 05:56 UTC   364d            etcd-ca                 noetcd-peer                  Dec 10, 2022 05:56 UTC   364d            etcd-ca                 noetcd-server                Dec 10, 2022 05:56 UTC   364d            etcd-ca                 nofront-proxy-client         Dec 10, 2022 05:56 UTC   364d            front-proxy-ca          noscheduler.conf             Dec 10, 2022 05:56 UTC   364d                                    noCERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGEDca                      Nov 17, 2031 14:53 UTC   9y              noetcd-ca                 Nov 17, 2031 14:53 UTC   9y              nofront-proxy-ca          Nov 17, 2031 14:53 UTC   9y              no

Post implementation steps

After running the kubeadm renew command we should restart the control plane Pods. Dynamic certificate reload is currently not supported for all components and certificates. Since,Static Pods are managed by the local kubelet and not by the API Server, thus kubectl cannot be used to delete and restart them.To restart a static Pod we can temporarily remove its manifest file from /etc/kubernetes/manifests/ and wait for 20 seconds.The kubelet will terminate the Pod if it’s no longer in the manifest directory.We can then move the file back and after another fileCheckFrequency period, the kubelet will recreate the Pod and the certificate renewal for the component can complete.

Managing Kubelet certificate renewal

Kubeadm configures a kubelet with automatic rotation of client certificates by using the /var/lib/kubelet/pki/kubelet-client-current.pem symlink specified in /etc/kubernetes/kubelet.conf. In case of failure in rotation process, we might see errors such as x509: certificate has expired or is not yet valid in kube-apiserver logs. To manage the manual renewal of kubelet certificates we can follow Kubernetes doc.