Grafana - forcing promtail to use hostname of the machine

Monitoring and Observability Grafana
#1

Hi, how to force promtail to use hostname of the machine? I tried -client.external-labels , but it does not work. It seems I am missing something :slightly_smiling_face:. Thanks for your help.

#2

Have you seen this thread? https://github.com/grafana/loki/issues/636

#3

First of all, I sorry I mentioned Loki, but I thought Promtail.
Yes, I saw it. But did not read it too carefully. I did it again. But I guess, I should use different way.
There is -config.expand-env=true and I can save the hostname in env variable. I did some test, hostname is stored in env variables, but promtail is not using it for instance in promtail.yml configuration file. I modified the service file to exec the command with the -config.expand-env=true and put ${hostname} to promtail.yml. But promtail is not pulling the hostname from the env variable.

#4

Are you deploying promtail with some sort of configuration management? I use ansible and set the hostname as a label that way

#5

Yes, I use ansible too. Before we did it via ansible too. Now we need to dynamically change the hostname and I am looking for the way to do it. The script which overwrite the hostname, can also overwrite the ansible yml. I am only looking for the way, how to do it without hardcoding the hostname to the yml config.

#6

Having only ever done it via ansible, it’s not something I have run into. Are you able to extract the hostname from a log line perhaps? I suppose you could set /etc/hostname as a scrape target :stuck_out_tongue:

#7

That wasn’t a particularly serious suggestion btw… let me also have a think about it

#8

I have made it as env variable, because of this article https://grafana.com/docs/loki/latest/clients/promtail/configuration/#use-environment-variables-in-the-configuration

#9

But I was not able to pull it via the config

#10

I tried ${HOSTNAME} etc., but no success. The promtail sees instance=""

#11

Can you show your promtail config where you set the label?

#12

Seems it’s not working for me either when i test now with -config.expand-env=true

#13

Yes, thats the point. I send the conf later today. I already erased the test device and doing fresh install.

#14

I tried also -config.expand-env or -config.expand-env true, but no luck

#15

I was setting it directly in the service file for promtail.

#16

I tested on a super old server with sysvinit, no luck

#17

It is supported on promtail version 2.1+

#18

I am trying it on fresh Archlinux with latest packages

#19

Promtail i was using is 2.2

#20

My try:

  journal:
    max_age: 12h
    labels:
      job: systemd-journal
      instance: ${HOSTNAME}
  relabel_configs:
  - source_labels: ['__journal__systemd_unit']
    target_label: 'unit'```
I tried it even with quotes etc.