Representing histogram data using heatmap, visualization confusing

Monitoring and Observability Grafana
1

Hello,
I am trying to represent the histogram data using Histogram or heatmap, somehow the visualization is not depicting the details and to be honest it’s confusing.
Metrics are in this format

controller_runtime_reconcile_time_seconds_bucket{controller="controller",le="0.01"}
...```


Query used is this


```group by (le, controller) (rate(controller_runtime_reconcile_time_seconds_bucket[$__rate_interval])```


I chose Heatmap as the panel type and also opted for `Format: Heatmap` and `Type: Range` in the options.
Do you suggest any other changes / configuration which needs to be done or should I explore panel like historgram? So far things I tried with histogram panel didnt cone out well.
2

Hmm yeah I dont know. I put up a prometheus query and was playing around with it here and it works basically what I’d expect https://play.grafana.org/d/feavrpzb4wu0wa/kristina-demos?orgId=1&from=now-30d&to=now&timezone=utc&var-fruit=apple&var-fuelType=Battery%20storage&var-stand=ice-king&var-employeeLastName=Swan&editPanel=29|https://play.grafana.org/d/feavrpzb4wu0wa/kristina-demos?orgId=1&from=now-30d&to=now&[…]ge&var-stand=ice-king&var-employeeLastName=Swan&editPanel=29 (feel free to edit anything, you just wont be able to save)

3

And your data contains values between 200 und 900 ms?

4

How do you interpret those visuals, its so confusing :slightly_smiling_face: .
With the images shown , how is it possible to interpret how many entries were there in which bucket?

5

Points were from 0.5ms to 60s

6

The colors indicate the “count”. purple in your case means “2” (but its weird that the legend shows values up to 1200 that I’m not seeing in the chart itself)

7

Prometheus histograms are usually cumulative, have you accounted for this?

8

For example,
I have these 2 data entries

controller_runtime_reconcile_time_seconds_bucket{controller="pxc-controller",le="1.5"} 8793```


It's easy to depict that reconcile taken between 1.25 to 1.5 seconds were 8793-3688.
I am not sure how can I derive that value in the graph
9

Prometheus histograms are usually cumulative, have you accounted for this?

You mean the entries in the bucket? Like if there are 2 entires ,
LE 1 and LE2.
LE2=LE1 + LE(1-2) ?

10

You have 3688 requests taking less tha 1.25 seconds and you have 8793 requests taking less than 1.5 seconds. That means you have 5105 requests taking between 1.25 and 1.5 seconds

11

I gather you have already looked at https://grafana.com/docs/grafana/latest/panels-visualizations/visualizations/heatmap/?

12

Lots of examples to be found in the playground: https://play.grafana.org/d/heatmap-calculate-log/grafana-heatmaps?orgId=1&from=now-6h&to=now&timezone=utc

13

Hi First of all , thank you for taking your time on weekend to help , I really appreciate it :slightly_smiling_face:
Yes I was wondering if there is a easy way to see 5105 in the graph panel.
When I used the heatmap.
It was just not giving the right visuals

14

Let me check the playgrounds once more

15

Ah, and I just noticed: why do you group by le AND controller? You should only group by le for a heatmap (X is time, Y is le, color is count – you cannot display a fourth dimension)

16

I have 3 different controllers.
like labels with controller=controller1, controller2, controller=controller3

17

It’s been a while since I configured a heatmap (and I admit it’s tricky). Let me quickly try myself before I continue giving half-baked tips

18

I need a way to represent all those controllers in a single heatmap and have the ability to switch between any controller in a single panel

19

I am not sure , its even possible , but tats what I am trying

20

I’m afraid that won’t sensibly be possible. You can add a dashboard variable to switch between controllers, but you can’t display them in a single panel at the same time