Configuring Prometheus alert state history

Monitoring and Observability Grafana
1

Question,
If I configure alert state history with promethues
https://grafana.com/docs/grafana/latest/alerting/set-up/configure-alert-state-history/

Does that state history not get saved anymore on the grafana end.
Since have configured to be stored now , on prometheus end?

2

Maybe I won’t answer this question fully, but when I was enabling alerting history with loki, you could specify if you want to write the history only to loki via feature toggle - I would guess the same goes here.

3

that helps,
Ill play with the feature toggles.

4

Alert State History saves Data in a Loki Backend or in a Prometheus Backend or both. It is Not saved in Grafana itself.

5

Thats are the Options in grafana.ini
[unified_alerting.state_history]
enabled = true
backend = multiple

primary = loki

URL of the Loki server.

loki_remote_url = http://localhost:3100|http://localhost:3100

secondaries = prometheus

Target data source UID for writing alert state changes.

prometheus_target_datasource_uid = <DATA_SOURCE_UID>

6

Feature toggles are Not needed anymore I think. However would probably Not Hurt as a Test.

7

So, when not enabled.
Where is the state history saved? When looking at alert rules.

As you can see the history of when things got fired.
My guess was in the DB backend .

Was noticiing that when I have that enabled for prometheus.

I was not able to view history in the alert rules.
So hence, my original question if then things just get stored on the backend you have enabled for?

8

by default state history is stored in annotations (in grafana database)

9

I was hoping if can keep annotations and as well the alert state history in both,
But again, unsure if that is posible

10

I have yet to play with configs.
So will tinker , when free up on my end

11

I think it should be fine → https://github.com/grafana/grafana/blob/2a5e8c8f9f4fbd84d9f749535487c7fca4fcf6fb/pkg/setting/setting_unified_alerting.go#L507 you can most probably split them with “,” looking at the code

12

perfect,
things make sense now. Ill setup with both annotations and prometheus . That should give me what I want.
Thanks again

Will see if have time EOD to tinker with that . But I see that should be good now.

13

that did it .

I followed docs for the setting up a primary and secondary

I just did annotations for primary and prometheus for secondary.

I’m now seeing the alert history like before via grafana UI end and as well get my metrics on prometheus end.
https://grafana.com/docs/grafana/latest/alerting/set-up/configure-alert-state-history/

Thanks again!

14

I will test more to conifrm things are working as expected.
But on quick glance . I think this did it .