Struggling to deploy Kubernetes pod with FailedMount warning in namespace

DevOps Kubernetes
1

Hey all,

When I deploy a simple pod to a namespace. kubernetes fails to deploy the pod with the below warning

│   Type     Reason       Age                 From               Message                                                                                                                                                                                                       │
│   ----     ------       ----                ----               -------                                                                                                                                                                                                       │
│   Normal   Scheduled    13m                 default-scheduler  Successfully assigned monitoring/helloworld-deployment-64d678f457-9t9bj to k8-worker                                                                                                                          │
│   Warning  FailedMount  50s (x14 over 13m)  kubelet            MountVolume.SetUp failed for volume "kube-api-access-fsmhg" : configmap "kube-root-ca.crt" not found```


However, running the same pod in the default namespace seems to work, I found some suggestions online to add `automountServiceAccountToken: false` to the pod spec, this allows the pod to be deployed but it's not a good enough solution since we can't access the API anymore, plus when using third-party helm charts I would rather not pull and modify them.

The question is, what is causing this, and how can I fix it.  I have been struggling with this issue for two days.

PS: The `kube-root-ca.crt` config map is deployed to the namespace
2

can you check if both pod and config-map are present in same namespace?

3

Yes, I can confirm they are

4

can you share the output of kubectl get all -n <YOUR NAMESPACE>?

5 
NAME                                         READY   STATUS              RESTARTS   AGE
pod/helloworld-deployment-64d678f457-9t9bj   0/1     ContainerCreating   0          22m

NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/helloworld-deployment   0/1     1            0           22m

NAME                                               DESIRED   CURRENT   READY   AGE
replicaset.apps/helloworld-deployment-64d678f457   1         1         0       22m```
6 
NAME               DATA   AGE
kube-root-ca.crt   1      10h```
7

Create a new Namespace and deploy it there. Does it work?

8

Please use code blocks rather than screenshots.

9

Thanks, will update with code blocks. Also, creating it in a new namespace doesn’t work either

10

just out of curiosity, are you trying to use ServiceAccount in your Pod?

11

It’s the default SA,

12

If it helps I am running k8s v1.28.5

13

are other pods within same NS experiencing the same issue?

14

can you create another SA, and use it in your deployment, see if it resolves?

15

Tried that, has the same error

16

maybe this can help https://github.com/kubernetes/kubernetes/issues/105204#issuecomment-1791824012

17

Thanks , but that’s not the case for me. I am using a very bare yaml file. Below is what I am trying to deploy

kind: Deployment
metadata:
  name: helloworld-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: helloworld
  template:
    metadata:
      labels:
        app: helloworld
    spec:
      serviceAccountName: test-sa
      containers:
      - name: helloworld
        image: nginxdemos/hello
        ports:
        - containerPort: 80```
18

The only thing that I can think of going through node’s Kubelet logs where Pod is getting scheduled. May be the issue is with something else.

19

I think I might have figured out why

20

Apparently, the admission plugin, NamespaceLifecycle , was not enabled on my api-server. I enabled it and restarted the api-server and it now works. I am not sure why it was working in the default namespace though, but it now works.