Setting up secure communication between Prometheus and Grafana Cloud effectively

Monitoring and Observability Grafana
#1

Hey guys, I am setting up monitoring & alerts for a webapp. I am looking at grafana cloud + prometheus (on local server), I have a few questions:

• To connect to prometheus I find remote write to be a good way, as it will work without opening ports :smile:
• However, I can’t understand how I can secure this set up to protect the communication between prometheus and grafana cloud
◦ Docs provide an example with basic auth, but is there a stronger alternative to that?
◦ The problem with basic auth is that the communication is still NOT encrypted
◦ Are there better ways to secure this - maybe something like mutual TLS ?
◦

#2

ASFAIK, every grafana cloud instance is an https/tls one. Since you’re using this HTTPS for the Prometheus remote write to Grafana cloud, the basic auth goes encrypted using the TLS of your grafana cloud instance.

#3

For some reference, here’s the snippet from the official docs:

`remote_write:

https://grafana.com/docs/grafana/latest/getting-started/get-started-grafana-prometheus/#get-started-with-grafana-and-prometheus|https://grafana.com/docs/grafana/latest/getting-started/get-started-grafana-prometheus/#get-started-with-grafana-and-prometheus

#4

Interesting, but the endpoint being https does not by itself guarantee encryption. The docs do say https, and also “basic auth” at the same time, so it’s confusing.

Also prometheus’ remote write config mentions a ‘tls_config’ section, but we aren’t using that here. Or maybe I’m just missing something very simple (not the first time).

https://prometheus.io/docs/prometheus/latest/configuration/configuration/#tls_config

#5

Examples that I find, that use TLS encryption, use these options to specify the certs to verify. Can’t find something like this for grafana cloud though.

#6

Ok, lets get back to the basics:

• First, on Prometheus docs on remote_write: what the basic_auth does is “Sets the Authorization header”.

```# Sets the Authorization header on every remote write request with the

configured username and password.

password and password_file are mutually exclusive.

basic_auth:
[ username: ]
[ password: ]
[ password_file: ]```

• Second, talking about Networking: if we use OSI Layering as reference, the HTTP Headers should be referenced on “Layer 6 - Application”, and TLS should be at “Layer 5 - Session”. Then we can conclude that the Headers are encrypted by TLS/HTTPS.

#7

About the certificates… When Grafana cloud create your instance, a common name certificate that already got a common name *.http://grafana.net|grafana.net as a certificate… and it’s referenced on the chain certificates of most of Operational Systems and cloud solutions. So, you don’t need to set-up the custom CAs or self-signed ones, like you need to do on On-Prem deployments.

#8

That being said, “please don’t believe me”, believe on the process, and do this:
• 1 - spin up your prometheus conf, sending it to your grafana cloud instance
• 2 - on the prometheus machine, do a tcpdump -vvvs 1024 -l -A host <http://yourname.grafana.net|yourname.grafana.net> and try to see the headers.

#9

Thanks for taking the time to explain all of this, definitely helps! Your comments make sense, I’ll try spin up a prometheus container and inspect traffic to grafana cloud. Will reply. Thanks agin for the detailed steps :smiley:

#10

Okay so here’s what I tried, I ended up using wireshark and used ip.dst == as a filter
The host filter didn’t work for me (maybe because those are not visible as the traffic is encrypted?)

With that I do see the usual pattern of a client hello and then Application Data being sent periodically…

When we follow the TCP conversation, we can see the ASCII encoded conversation which is gibbrish

I am not an expert, but I guess that’s proof enough that the connection is encrypted :smiley:

#11

If you want to see the unencrypted version, spin up a Grafana instance in plain HTTP as a neighbor of your app, and try to remote write to this plain HTTP instance, and you’ll see the plain headers on the tcpdump/wireshark