How can we get Grafana metrics in JSON or any other format using any CLI tool

Monitoring and Observability Prometheus
#1

Hi there,
I have a test case, wherein I have to deploy Prometheus to capture the pods and node-specific metrics, such as metrics exposes by node exporter and Kube-state-metric. The servers are very isolated with no connectivity to the web UI or any dashboard. Need some help to understand, how can we get these metrics in JSON or any other format using any CLI tool? Because of the environment restriction of no access to the web UI or dashboards, cannot deploy Grafana. Your suggestions will be a huge help.

I query API using "kubectl get --raw /apis/metrics.k8s.io/v1beta1/pods | jq ‘.’ but I can just see the memory and CPU raw data, what should I add to get more metrics in this case? For example, metrics exposes by kube-state-metrics and node-exporter which includes more data points such as pods state/health, network, disk usage,.etc,

Thanks,

#2

Your message is absolutely confusing. E.g.
• what are these question marks?
• what’s the question
Besides of that:
• have you checked https://prometheus.io/docs/prometheus/latest/querying/api/#targets ?
• do you see any targets using GET /api/v1/targets?
• is kubectl port-forward an option?

#3

How did you get the “CPU raw data”?

#4

Hello, thanks for responding. My bad for the typo’s, i have updated the query now.

#5

got the cpu and memory data using

"“kubectl get --raw /apis/metrics.k8s.io/v1beta1/pods | jq '.”

#6

@UDBJM0RRS well how can i use the port-forward to capture the raw data on the CLI terminal? I cannot access the browser UI ? I am new here, any advise is highly appreciated

#7

user@gateway10:~$ kubectl get --raw /apis/metrics.k8s.io/v1beta1/nodes | jq '.'
{
"kind": "NodeMetricsList",
"apiVersion": "[metrics.k8s.io/v1beta1](http://metrics.k8s.io/v1beta1)",
"metadata": {
"selfLink": "/apis/metrics.k8s.io/v1beta1/nodes"
},
"items": [
{
"metadata": {
"name": "k3s-is2",
"selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/ah-ifc2",
"creationTimestamp": "2021-03-09T09:34:29Z"
},
"timestamp": "2021-03-09T09:33:52Z",
"window": "30s",
"usage": {
"cpu": "584472617n",
"memory": "5311356Ki"
}
}
]
}

#8

With this command you get the metrics reported through Kubernetes - you would get them without Prometheus.

to be honest, it’s hard to propose a solution here, as there are foundational gaps in terms of understanding of Kubernetes and Prometheus. Maybe it’s worth to take a step back and getting familiar with both components and their role in a typical setup. I haven’t read through this article in detail, but it looks complete (don’t just look at the pictures!)
https://sysdig.com/blog/kubernetes-monitoring-prometheus/