Representing histogram data using heatmap, visualization confusing

I mean, what’s your expected output? How would you distinguish between your controllers in the heatmap?

Like how I have the option of selecting specific variable in a time series, I can click one of the options and the time series shows only the required stuff

I can easily switch between go routines and threads data, by clicking the item in legend

yeah, you can’t really do that with a heatmap. heatmaps only allow displaying a single series

check this: https://grafana.com/blog/2020/06/23/how-to-visualize-prometheus-histograms-in-grafana/

You want to use a query similar to:

sum by (le) (increase(controller_runtime_reconcile_time_seconds_bucket[$__rate_interval])

(increase is more or less absolute, rate gives you the per-second values; it depends on your use case what you want to show. The heatmap will look the same, but the values will be different)

Oh ok, So if I want to have data of all the controllers , only way might have to be using different panel for different controllers

Yes. But look into “<https://grafana.com/docs/grafana/latest/panels-visualizations/configure-panel-options/#configure-repeating-panels|Repeat options>” of the panel. You can define a variable and have Grafana generate one panel per variable value. I used it on a past project to display one heatmap per entity.

There’s also a blog post with a walkthrough: https://grafana.com/blog/2020/06/09/learn-grafana-how-to-automatically-repeat-rows-and-panels-in-dynamic-dashboards/

I will use repeat options, I should potentially use group by rite if I want to have data for individual bucket rite

something like

group by (le) (increase(controller_runtime_reconcile_time_seconds_bucket)

You want to use sum to aggregate all dimensions into a single value

sum by (le) (increase(controller_runtime_reconcile_time_seconds_bucket{controller=~"$your_dashboard_variable"}[$__rate_interval])

And then repeat the panel based on $your_dashboard_variable

ah ok,
Getting something like this.
So looking much better. WIll try with few combinations