Anyone good at reading docker logs for prometheus. Today my prometheus docker instance just stop allowing connections to TCP 9090. I’ve rebuilt it all and it does the same thing. After starting up docker and running prometheus it all works, then it stops and I can’t even curl http://ip:9090. What is interesting is if I change the servers IP it’s stable, but I need to keep it on the original IP address. I think something is spamming the port (our own DDOS). If I look at the logs for prometheus I see these errors as soon as it stops working, 100s of them.
time=2025-06-17T19:50:53.136Z level=ERROR source=write_handler.go:161 msg="Error decoding remote write request" component=web err="read tcp 172.18.0.2:9090->10.10.38.114:58733: i/o timeout"
time=2025-06-17T19:50:53.362Z level=ERROR source=write_handler.go:161 msg="Error decoding remote write request" component=web err="read tcp 172.18.0.2:9090->10.10.38.22:57699: i/o timeout"
time=2025-06-17T19:50:53.367Z level=ERROR source=write_handler.go:161 msg="Error decoding remote write request" component=web err="read tcp 172.18.0.2:9090->10.10.38.22:57697: i/o timeout"
time=2025-06-17T19:50:53.367Z level=ERROR source=write_handler.go:161 msg="Error decoding remote write request" component=web err="read tcp 172.18.0.2:9090->10.10.38.88:51980: read: connection reset by peer"
time=2025-06-17T19:50:53.613Z level=ERROR source=write_handler.go:161 msg="Error decoding remote write request" component=web err="read tcp 172.18.0.2:9090->10.10.38.114:59295: read: connection reset by peer"```
10.10.38.0/24 is a test network which is have network issues, there are devices on there with alloy sending to the prometheus server. I can't get on the network to stop these or get hold of anyone to troubleshoot as the site is closed. I'm hoping it is this site as I've changed nothing and can't think of any reason why Prometheus is having issues. In docker is shows as up and healthy, but I think TCP 9090 is being blocked be this traffic.
I tried a local fw rule on Ubuntu to block 10.10.38.0/24 inbound and outbound, but I still get these errors above. Any suggestions would be great.