My Prometheus is unable to scrape metrics, any suggestions?
Caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:283: Failed to list *v1.Pod: pods is forbidden: User “system:serviceaccount:canary-analysis:prometheus-server” cannot list resource “pods” in API group “” at the cluster scope" is one of the errors that continues to repeat
Also trying to use Rancher monitoring which uses prometheus but there are not logs, all It displays is : nginx: [warn] duplicate MIME type “text/html” in /var/cache/nginx/nginx.conf:45
Service account which you are created for prometheus do not have authorization to list resources. you can check what level of access service account has. you can refer below one:
kind: ClusterRole metadata: name: prometheus rules: - apiGroups: [""] resources: - nodes - nodes/proxy - services - endpoints - pods verbs: ["get", "list", "watch"] - apiGroups: - extensions resources: - ingresses verbs: ["get", "list", "watch"] - nonResourceURLs: ["/metrics"] verbs: ["get"] --- apiVersion: <http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1> kind: ClusterRoleBinding metadata: name: prometheus roleRef: apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io> kind: ClusterRole name: prometheus subjects: - kind: ServiceAccount name: default namespace: monitoring```