Configuring Grafana timeouts for proxy connections in Ambassador using Nginx

Monitoring and Observability Grafana
#1

Hi All,

I am getting this while querying in Grafana, is it coming from loki ?. If yes then where I need to add timeout for proxy and other things. We are using ambassador there we have 300 seconds timeout. When I searched this I get to know we need to add these three flag in nginx conf. But I am not getting where to add, because there are several routes in values.yaml
proxy_connect_timeout 600s;
proxy_send_timeout 600s;
proxy_read_timeout 600s;

#2

This seems to me like a network or firewall issue. Is this running locally or in the cloud and also how did you deploy it? helm chart /manually?

#3

If its deployed via helm chart it would also be helpful to know if loki is in the same cluster and namespace

#4

Yes it is deployed as helm charts and both are in same namespace.

#5

It is in AWS EKS cluster.

#6

I would try to shell into the grafana pod and try to connect to the url you are getting the 504 on typically the pod will have something like wget or telnet or ping.
If you cant connect that way then try with the internal dns name within your cluster. You most likely need to whitelist the ip or ip range of the loki pod on your security group for the cluster inbound.

#7

depending on how it is connecting it will either be the internal network cidr or the external ip being used by the loki pod