Sending logs from Grafana cloud to Loki

Monitoring and Observability Grafana
#1

Good morning, I’ve managed to get Prometheus metrics through to my Grafana Cloud by using the Grafana Agent instead of standard Prometheus. Now I want to get logs into Loki. Is there a custom agent for that as well?

#2

Good morning! Grafana Cloud Agent can also send logs to your Loki instance in the Cloud. If you’re using k8s, then you can take a look at the manifest (https://github.com/grafana/agent/blob/master/production/kubernetes/agent-loki.yaml), installation script (https://github.com/grafana/agent/blob/master/production/kubernetes/install-loki.sh) or general k8s docs - https://github.com/grafana/agent/tree/master/production/kubernetes.

Here’s a bit more info regarding sending logs - https://github.com/grafana/agent/blob/master/docs/getting-started.md#loki-configmigrating-from-promtail.

#3

Aha cool, so it’s just a matter of extending the config yaml?

#4

I want to try it on my dev box first

#5

Yeah, it’s just the matter of configuration :slightly_smiling_face:

#6

Ok but the agent currently scrapes the metrics by calling the /metrics endpoint on the server. How does it get the logs?

#7

I’m using slf4j with logback

#8

Also, this config looks different to what the wizard generates

#9

are you saying the agent has promtail inside it?

#10
  • mind me asking - which wizard are you refering to ?
#11

The one in the lightning menu

#12

Walkthrough, it’s called

#13

Ok from what I can read, I need logback to write a file and I simply tell the agent config what the file is, and it ships it to the cloud?

#14

the agent uses the same logic as promtail, yes, albeit the config is slightly different. As far as promtail configuration goes, you can check out https://grafana.com/docs/loki/latest/getting-started/get-logs-into-loki/ :slightly_smiling_face:

#15

Tl;dr you need to figure out proper promtail config for your use-case and then add it to grafana cloud agent config (possibly with minor adjustments).

#16

Now I’m more confused. I already have a working Prometheus metrics config that uses the agent. I can add a section and that will use the agent’s built-in promtails?

#17

> I can add a section and that will use the agent’s built-in promtails?
Yes.

#18

So in your agent config you have a prometheus section. To get the agent to scrape logs, you’d need to add a loki section. Here’s an example from the docs (https://github.com/grafana/agent/blob/master/docs/getting-started.md):

  configs:
  - name: default
    positions:
      filename: /tmp/positions.yaml
    scrape_configs:
      - job_name: varlogs
        static_configs:
          - targets: [localhost]
            labels:
              job: varlogs
              __path__: /var/log/*log
    clients:
      - url: [http://localhost:3100/loki/api/v1/push](http://localhost:3100/loki/api/v1/push)```
#19

Cool thanks. Sorry I was away earlier. I’ll check it out.

#20

My main question is the URL it pushes to. Should that be replaced by some grafana cloud URL?