Using Grafana managed alerts with external Prometheus Alertmanager datasource

Monitoring and Observability Grafana
#1

Using the Grafana provisioning API, we’re running several stateless Grafana instances where we build our own images based on the public grafana image that provision all of our datasources, dashboards, and alerts from text files. The problem I’m seeing however, is using Grafana managed alerts, with an external Prometheus Alertmanager datasource, in a stateless container, the builtin Grafana Alertmanager keeps enabling itself on container restarts. Is there any way to use the provisioning file API to completely disable the builtin alertmanager, but still use Grafana managed alerts?

#2

The actual problem I’m trying to solve, is that when the container starts up, the Grafana alertmanager is re-enabled (but unconfigured.) So all of our Grafana managed alerts start going to an unconfigured builtin Alertmanager, and thus go nowhere.

#3

For now, I’ve temporarily solved this by just adding a persistent volume for the grafana sqlite database. But that’s undesirable because we’re only keeping that database around between container restarts for this one very small issue, so ideally we could just use the provisioning API to disable the builtin Alertmanager.

Or alternatively maybe there’s some other way of pinning our Grafana managed alerts to our external Prometheus Alertmanager datasource without disabling the builtin Alertmanager, but that doesn’t appear to be the case looking at the docs.

#4

in your alertmanager datasource configuration, are you setting handleGrafanaManagedAlerts: true

#5

(I’m not using grafana managed alerts so I’m not really sure here)

#6 
          type: alertmanager
          url: <http://alertmanager-main.prometheus.svc:9093>
          access: proxy
          jsonData:
            implementation: prometheus
            handleGrafanaManagedAlerts: true```


That's our alertmanager datasource config.
#7

Appreciate you looking anyway. But yeah, despite that bit of config, the builtin alertmanager takes precedence, and needs to be disabled in the UI under Alerting > Settings in the left side bar before all of our alerts start going to our provisioned Prometheus Alertmanager datasource

#8

This is also on Grafana 11.1.0, by the way. Which I believe is latest