Renewing all certificates in a cluster causing pod recreation and cluster-info error

DevOps Kubernetes
#1

Hi all, i have a cluster where all certificate expired yesterday

so this morning i renew all certificate using command kubeadm certs renew all on the master, after that the apiserver and other component started normally.
but now when i try to delete or update a resource the pod is recreated and when i delete a pod in a deployment procedure it is not recreated, i also tried to add a new node to the cluster and got this error [discovery] The cluster-info ConfigMap does not yet contain a JWS signature for token ID "xeyhmo", will try again

please can anyone help ? thanks in advance.

#2

hi. this might indicate problems with kube-controller-manager.
make sure you restarted all components including the KCM once “certs renew” finished. KCM needs only its kubeconfig file, the CA.key/CA.cert as well.

the JWS error means that the KCM is updating a particular token ID. if that is a valid / non-expired token, then something is not OK at the KCM.

the pod deletion problems could indicate that something is wrong in e.g. the replicaset controller at the KCM.

check if KCM is crasdhing/restarting and look at its logs.

#3

hello this is error i am getting on KCM
error retrieving resource lock kube-system/kube-controller-manager: Unauthorized

#4

try kubeadm certs renew controller-manager.conf and then restart the KCM.

#5

how do i restart KCM please

#6

https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/

After running the command you should restart the control plane Pods. This is required since dynamic certificate reload is currently not supported for all components and certificates. Static Pods are managed by the local kubelet and not by the API Server, thus kubectl cannot be used to delete and restart them. To restart a static Pod you can temporarily remove its manifest file from /etc/kubernetes/manifests/ and wait for 20 seconds (see the fileCheckFrequency value in KubeletConfiguration struct. The kubelet will terminate the Pod if it’s no longer in the manifest directory. You can then move the file back and after another fileCheckFrequency period, the kubelet will recreate the Pod and the certificate renewal for the component can complete.

#7

but just move the KCM manifest file, not the rest.

#8

but if you have not done this after the initial “certs renew” that you did you can move all files.

#9

moving files temporary = restarts components

#10

thanks very much , it was exactly what i have to do