Unable to scrape Prometheus metrics

Monitoring and Observability Prometheus
#1

My Prometheus is unable to scrape metrics, any suggestions?

#2

Caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:283: Failed to list *v1.Pod: pods is forbidden: User “system:serviceaccount:canary-analysis:prometheus-server” cannot list resource “pods” in API group “” at the cluster scope" is one of the errors that continues to repeat

#3

Also trying to use Rancher monitoring which uses prometheus but there are not logs, all It displays is : nginx: [warn] duplicate MIME type “text/html” in /var/cache/nginx/nginx.conf:45

#4

Service account which you are created for prometheus do not have authorization to list resources. you can check what level of access service account has. you can refer below one:

kind: ClusterRole
metadata:
  name: prometheus
rules:
- apiGroups: [""]
  resources:
  - nodes
  - nodes/proxy
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"]
- apiGroups:
  - extensions
  resources:
  - ingresses
  verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]
---
apiVersion: [rbac.authorization.k8s.io/v1beta1](http://rbac.authorization.k8s.io/v1beta1)
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: [rbac.authorization.k8s.io](http://rbac.authorization.k8s.io)
  kind: ClusterRole
  name: prometheus
subjects:
- kind: ServiceAccount
  name: default
  namespace: monitoring```