Grafana 10.1.2 alerts issue: Random 'invalid service state' error

hey everyone, i have a grafana version 10.1.2
we are using the alerts in the grafana, and for some reason we are getting an error(not always, its random)
Error: ✗ not healthy, 0 terminated, 1 failed: [invalid service state: Failed, expected: Running, failure: contact points: could not find object using provided id and hash]
we creating the alerts with configmap and yaml.
what can be the problem? does anyone had this issue?

Did you search on GitHub? And how is your Grafana installed ? (Multiple instances, Kubernetes, from binaries, etc)

yes i did, it doesn’t help.
i am using 10.2 version not even 9.something
we are using grafana in k8s cluster, 3 instances in HA,
i made sure that there is no contact point that have any problem or that doesn’t exists.
we are provisioning our alerts with config files that contain yaml files that are been mounted to the pods.

Did you used our Helm chart to deploy or is it custom ?

I know there is some fuss to handle about alerting with HA Grafana (as you don’t want to end up with 3 instances firing the same alert at the same time)

Our Helm chart is supposed to expose options for that.

yes i used your helm chart

maybe i need to enable some flag?

Probably. I honestly never used a multiple instances deployment myself but I guess this could be related to that.

ok i will dig a bit into it:pray: