So when I enable Grafana 8 alerts, a bunch of predefined rules are loaded from prometheus configuration. I guess this is more of a prometheus question, but how would I delete these predefined rules?
They are not loaded to Grafana. The alerting page just displays them. Unfortunately, it does not support managing Prometheus alerts. just read only for now unless you use Cortex\Loki.
You can remove those alerts from the page by just turning off support for ruler API. You can find that switch in Prometheus data source
Well that was easy, thanks a ton!