Grafana - No data while querying for errors on my http_routes

Monitoring and Observability Grafana
#1

Hi I’m trying to query for errors on my http_routes but I don’t get any data. I think there’s an issue with my query on escaping.

sum by(http_route, http_method) (increase(http_server_duration_milliseconds_count{http_route=~`${http_route:regex}`, http_method=~"${http_method}",http_status_code=~"5..", hostname=~"${hostname}"}[$__rate_interval]))

Once I delete http_route param I get the data.

I have tried with

http_route=~`${http_route:regex}` http_route=~`${http_route}` http_route=~"${http_route}" http_route=~`${http_route:regex}

`
What am I missing :confused:

#2

Have you tried to see in query explorer what is going out to your datasource? I’d check if the variable is named http_route or if it’s only just a label (could happen that the name is different than the label). Also make sure there is http_route label on your metric - if there isn’t, there won’t be any match. Variables should work though.

#3

I did, http_route is the correct one.

It’s when I select All (~970 routes) I don’t get any data and when I checked the payload in the netwok tab, the
route /* is send as /\\\\*

#4

Can you share the definition of the http_route variable? I’m pretty sure there should be .* as all. On top of that - do you receive any results when you pick a single route from variable?

#5

when I pick a single route yes.

I think it resolves to /\\\\* because I have a multi value template variale and I just select All

#6

as I don’t want to use .* to allow other teams to check their endpoints etc

#7

I don’t think I follow. Correct me if I’m wrong but you have a custom variable, which has a custom All value set to /* , right?

#8

no, I set http_route variable in the dashboard to All
One of the endpoints is /* and in the payload that’s sent out to Mimir I see that this route is changed to /\\\\*.

But I just checked my dashboard that has this query and it shows data for all …

sum by(http_route, http_method) (increase(http_server_duration_milliseconds_count{http_route=~"${http_route}", http_method=~"${http_method}",http_status_code=~"5..", hostname=~"${hostname}"}[$__rate_interval]))

Beforehand I set the Prometheus type (in DS config) to Mimir and set the version >2.9

#9

strange but I think it resolved it