Sending logs from Grafana cloud to Loki

Monitoring and Observability Grafana
1

Good morning, I’ve managed to get Prometheus metrics through to my Grafana Cloud by using the Grafana Agent instead of standard Prometheus. Now I want to get logs into Loki. Is there a custom agent for that as well?

2

Good morning! Grafana Cloud Agent can also send logs to your Loki instance in the Cloud. If you’re using k8s, then you can take a look at the manifest (https://github.com/grafana/agent/blob/master/production/kubernetes/agent-loki.yaml), installation script (https://github.com/grafana/agent/blob/master/production/kubernetes/install-loki.sh) or general k8s docs - https://github.com/grafana/agent/tree/master/production/kubernetes.

Here’s a bit more info regarding sending logs - https://github.com/grafana/agent/blob/master/docs/getting-started.md#loki-configmigrating-from-promtail.

3

Aha cool, so it’s just a matter of extending the config yaml?

4

I want to try it on my dev box first

5

Yeah, it’s just the matter of configuration :slightly_smiling_face:

6

Ok but the agent currently scrapes the metrics by calling the /metrics endpoint on the server. How does it get the logs?

7

I’m using slf4j with logback

8

Also, this config looks different to what the wizard generates

9

are you saying the agent has promtail inside it?

10
  • mind me asking - which wizard are you refering to ?
11

The one in the lightning menu

12

Walkthrough, it’s called

13

Ok from what I can read, I need logback to write a file and I simply tell the agent config what the file is, and it ships it to the cloud?

14

the agent uses the same logic as promtail, yes, albeit the config is slightly different. As far as promtail configuration goes, you can check out https://grafana.com/docs/loki/latest/getting-started/get-logs-into-loki/ :slightly_smiling_face:

15

Tl;dr you need to figure out proper promtail config for your use-case and then add it to grafana cloud agent config (possibly with minor adjustments).

16

Now I’m more confused. I already have a working Prometheus metrics config that uses the agent. I can add a section and that will use the agent’s built-in promtails?

17

> I can add a section and that will use the agent’s built-in promtails?
Yes.

18

So in your agent config you have a prometheus section. To get the agent to scrape logs, you’d need to add a loki section. Here’s an example from the docs (https://github.com/grafana/agent/blob/master/docs/getting-started.md):

  configs:
  - name: default
    positions:
      filename: /tmp/positions.yaml
    scrape_configs:
      - job_name: varlogs
        static_configs:
          - targets: [localhost]
            labels:
              job: varlogs
              __path__: /var/log/*log
    clients:
      - url: [http://localhost:3100/loki/api/v1/push](http://localhost:3100/loki/api/v1/push)```
19

Cool thanks. Sorry I was away earlier. I’ll check it out.

20

My main question is the URL it pushes to. Should that be replaced by some grafana cloud URL?