Fluent-bit for sending kubernetes pods logs to Graylog

DevOps
1

Hi, does anyone use fluent-bit to send kubernetes pods logs to graylog ? Can you share your config (SERVICE,INPUT, FILTER, PARSER, OUTPUT)? I’m getting logs from all the pods but not from nginx ingress pod for some reason.

2

Just throwing ideas to the table: does the ingress nginx pod have the logs in the same path as fluentbit expects?

3

Good question. When I do kubectl logs nginx-ingress-controller-pod I can see the logs.
My fluent-bit input is configured with:

    [INPUT]
      Name tail
      Path /var/log/containers/*.log
      Tag kube.*
      Parser docker
      DB /var/log/flb_kube.db
      Mem_Buf_Limit 5MB
      Skip_Long_Lines On
      Refresh_Interval 10```
4

What I meant was: if you ssh into nginx-controller-pod , are the logs in the path /var/log/containers ?

5

Because if that folder does not have *.log files, then fluent-bit won’t fetch them.

6

Hi, thats what I get

total 0
drwxr-xr-x 1 www-data www-data 23 Nov 23  2018 .
drwxr-xr-x 1 www-data www-data 19 Nov 23  2018 ..
lrwxrwxrwx 1 root     root     11 Nov 23  2018 access.log -> /dev/stdout
lrwxrwxrwx 1 root     root     11 Nov 23  2018 error.log -> /dev/stderr```
7

So your nginx logs are under var/log/nginx, not var/log/containers

Can you try to create a new input (or change the current one just for testing) with
Path /var/log/nginx/*.log

8

Yes, thank you, that makes sense. So can I have two inputs like that ?

      Name tail
      Path /var/log/nginx/*.log
      Tag kube.*
      Parser docker
      DB /var/log/flb_kube.db
      Mem_Buf_Limit 5MB
      Skip_Long_Lines On
      Refresh_Interval 10
[INPUT]
      Name tail
      Path /var/log/containers/*.log
      Tag kube.*
      Parser docker
      DB /var/log/flb_kube.db
      Mem_Buf_Limit 5MB
      Skip_Long_Lines On
      Refresh_Interval 10```
9

I’m not sure if you can have the same tag, but try it to see

10

Although now I was having a look at kubernetes logging architecture (https://kubernetes.io/docs/concepts/cluster-administration/logging/) and maybe that PATH is the path in the node where logs are stored instead of in the container.

tl;dr: I’m confused :sweat_smile:

11

But try that and let’s see

12

So. Fluent-bit accepts those inputs with below error:

[2022/01/27 11:08:09] [ warn] [input:tail:tail.1] error scanning path: /var/log/nginx/*.log```
The issue is that the fluent-bit looks for logs on the node where the pod is running.

On the node itself this folder exist: /var/log/containers

Inside I have those for each pod:
```nginx-ingress-controller-xxx-pod.log -> /var/log/pods/default_nginx-ingress-controller-xxx/nginx-ingress-controller/0.log```
I can do sudo cat both and get logs fine.

I will enable debug logs on fluent-bit to see
13

I restarted the pods a few times to check debug logs and started working…

14

At least I understand fluent-bit a little better

15

So the /var/log/containers folder is inside the node, not the pod? Makes sense i guess

16

You restarted fluent-bit and it just started to work?

17

I changed the log option to debug, then I started getting logs from nginx. then changed back to info and they are still working.

18

No idea, some bug maybe. Could be nginx, I need to update that too.

19

Maybe re-deploying fluent-bit did the trick

20

https://media4.giphy.com/media/FspLvJQlQACXu/giphy.gif