Certificate renewal for client certificates becomes usual process for a long running Kubernetes clusters. This blog is an implementation of the process to perform manual cert upgrade. There might be multiple business reasons to perform manual cert rotation due to stringent business requirements. For the Kubernetes clusters installed with kubeadm, all the client certificates generated by kubeadm expire after 1 year. As a standard procedure, manual renewal is not usually required as cert rotation gets performed automatically when we perform Kubeadm upgrade.
For scenarios to perform manual certificate renewal, during the last month or week of expiry period, we can use Kubeadm commands to verify the certs and renew them for the cluster. The operations team can connect to control plane node and access the kubeadm utility to manage the certificates.
Kubeadm Cert management
Kubeadm is equipped with cert management and renewal commands.
[root@kmaster ~]# kubeadm certs
Commands related to handling kubernetes certificates
Usage:
kubeadm certs [command]
Available Commands:
certificate-key Generate certificate keys
check-expiration Check certificates expiration for a Kubernetes cluster
generate-csr Generate keys and certificate signing requests
renew Renew certificates for a Kubernetes clusterKubeadm command for renew
Usage:
kubeadm certs renew [flags]
kubeadm certs renew [command]
Available Commands:
admin.conf Renew the certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself
all Renew all available certificates
apiserver Renew the certificate for serving the Kubernetes API
apiserver-etcd-client Renew the certificate the apiserver uses to access etcd
apiserver-kubelet-client Renew the certificate for the API server to connect to kubelet
controller-manager.conf Renew the certificate embedded in the kubeconfig file for the controller manager to use
etcd-healthcheck-client Renew the certificate for liveness probes to healthcheck etcd
etcd-peer Renew the certificate for etcd nodes to communicate with each other
etcd-server Renew the certificate for serving etcd
front-proxy-client Renew the certificate for the front proxy client
scheduler.conf Renew the certificate embedded in the kubeconfig file for the scheduler manager to useConfig and certs backup
For precautionary measure, it’s advised to perform backup for Kubernetes configs and certificates.
# Backup certs
mkdir -p $HOME/k8scluster-old-certs/pki
/bin/cp -p /etc/kubernetes/pki/*.* $HOME/k8scluster-old-certs/pki
ls -l $HOME/k8scluster-old-certs/pki/
#Backup configs
/bin/cp -p /etc/kubernetes/*.conf $HOME/k8scluster-old-certs
ls -ltr $HOME/k8scluster-old-certs
#Backup local config
mkdir -p $HOME/k8scluster-old-certs/.kube
/bin/cp -p ~/.kube/config $HOME/k8scluster-old-certs/.kube/.
ls -l $HOME/k8scluster-old-certs/.kube/.Certificate Expiry Check
We can check the cert expiry period with follow command :
[root@kmaster ~]# kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Nov 19, 2022 14:53 UTC 344d no
apiserver Nov 19, 2022 14:53 UTC 344d ca no
apiserver-etcd-client Nov 19, 2022 14:53 UTC 344d etcd-ca no
apiserver-kubelet-client Nov 19, 2022 14:53 UTC 344d ca no
controller-manager.conf Nov 19, 2022 14:53 UTC 344d no
etcd-healthcheck-client Nov 19, 2022 14:53 UTC 344d etcd-ca no
etcd-peer Nov 19, 2022 14:53 UTC 344d etcd-ca no
etcd-server Nov 19, 2022 14:53 UTC 344d etcd-ca no
front-proxy-client Nov 19, 2022 14:53 UTC 344d front-proxy-ca no
scheduler.conf Nov 19, 2022 14:53 UTC 344d no
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Nov 17, 2031 14:53 UTC 9y no
etcd-ca Nov 17, 2031 14:53 UTC 9y no
front-proxy-ca Nov 17, 2031 14:53 UTC 9y noThe above command shows the expiration/residual time for the client certificates. These certificates are located at /etc/kubernetes/pki on the control plane nodes of Kubernetes cluster.
NOTEkubelet.conf is not included in the list above because kubeadm configures
kubelet for automatic certificate renewal with rotatable certificates under
/var/lib/kubelet/pki. Certs renew uses the existing certificates as the
authoritative source for attributes (Common Name, Organization, SAN, etc.)
instead of the kubeadm-config ConfigMap. It is strongly recommended to keep
them both in sync.
Implementation of manual certificate renewal
We can renew the certificates manually at any time with the kubeadm certs renew command. This command performs the renewal using CA certificate and key stored in /etc/kubernetes/pki.
For an HA Kubernetes cluster,
kubeadm certs renewcommand needs to be executed on all the control-plane nodes.
Renew single certificate
[root@kmaster kubernetes]# kubeadm certs renew admin.conf
[renew] Reading configuration from the cluster...
certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewedResult of above step
[root@kmaster kubernetes]# kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Dec 10, 2022 05:53 UTC 364d no
.......Renew all certificates
[root@kmaster kubernetes]# kubeadm certs renew all
[renew] Reading configuration from the cluster...
certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed
Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.Result of above step
[root@kmaster kubernetes]# kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Dec 10, 2022 05:56 UTC 364d no
apiserver Dec 10, 2022 05:56 UTC 364d ca no
apiserver-etcd-client Dec 10, 2022 05:56 UTC 364d etcd-ca no
apiserver-kubelet-client Dec 10, 2022 05:56 UTC 364d ca no
controller-manager.conf Dec 10, 2022 05:56 UTC 364d no
etcd-healthcheck-client Dec 10, 2022 05:56 UTC 364d etcd-ca no
etcd-peer Dec 10, 2022 05:56 UTC 364d etcd-ca no
etcd-server Dec 10, 2022 05:56 UTC 364d etcd-ca no
front-proxy-client Dec 10, 2022 05:56 UTC 364d front-proxy-ca no
scheduler.conf Dec 10, 2022 05:56 UTC 364d no
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Nov 17, 2031 14:53 UTC 9y no
etcd-ca Nov 17, 2031 14:53 UTC 9y no
front-proxy-ca Nov 17, 2031 14:53 UTC 9y noPost implementation steps
After running the kubeadm renew command we should restart the control plane Pods. Dynamic certificate reload is currently not supported for all components and certificates. Since,Static Pods are managed by the local kubelet and not by the API Server, thus kubectl cannot be used to delete and restart them.To restart a static Pod we can temporarily remove its manifest file from /etc/kubernetes/manifests/ and wait for 20 seconds.The kubelet will terminate the Pod if it’s no longer in the manifest directory.We can then move the file back and after another fileCheckFrequency period, the kubelet will recreate the Pod and the certificate renewal for the component can complete.
Managing Kubelet certificate renewal
Kubeadm configures a kubelet with automatic rotation of client certificates by using the /var/lib/kubelet/pki/kubelet-client-current.pem symlink specified in /etc/kubernetes/kubelet.conf. In case of failure in rotation process, we might see errors such as x509: certificate has expired or is not yet valid in kube-apiserver logs. To manage the manual renewal of kubelet certificates we can follow Kubernetes doc.