Hi, does anyone use fluent-bit to send kubernetes pods logs to graylog ? Can you share your config (SERVICE,INPUT, FILTER, PARSER, OUTPUT)? I’m getting logs from all the pods but not from nginx ingress pod for some reason.
Just throwing ideas to the table: does the ingress nginx pod have the logs in the same path as fluentbit expects?
Good question. When I do
kubectl logs nginx-ingress-controller-pod I can see the logs.
My fluent-bit input is configured with:
[INPUT] Name tail Path /var/log/containers/*.log Tag kube.* Parser docker DB /var/log/flb_kube.db Mem_Buf_Limit 5MB Skip_Long_Lines On Refresh_Interval 10```
What I meant was: if you ssh into
nginx-controller-pod , are the logs in the path
Because if that folder does not have
*.log files, then fluent-bit won’t fetch them.
Hi, thats what I get
total 0 drwxr-xr-x 1 www-data www-data 23 Nov 23 2018 . drwxr-xr-x 1 www-data www-data 19 Nov 23 2018 .. lrwxrwxrwx 1 root root 11 Nov 23 2018 access.log -> /dev/stdout lrwxrwxrwx 1 root root 11 Nov 23 2018 error.log -> /dev/stderr```
So your nginx logs are under
Can you try to create a new input (or change the current one just for testing) with
Yes, thank you, that makes sense. So can I have two inputs like that ?
Name tail Path /var/log/nginx/*.log Tag kube.* Parser docker DB /var/log/flb_kube.db Mem_Buf_Limit 5MB Skip_Long_Lines On Refresh_Interval 10 [INPUT] Name tail Path /var/log/containers/*.log Tag kube.* Parser docker DB /var/log/flb_kube.db Mem_Buf_Limit 5MB Skip_Long_Lines On Refresh_Interval 10```
I’m not sure if you can have the same tag, but try it to see
Although now I was having a look at kubernetes logging architecture (https://kubernetes.io/docs/concepts/cluster-administration/logging/) and maybe that
PATH is the path in the node where logs are stored instead of in the container.
tl;dr: I’m confused
But try that and let’s see
So. Fluent-bit accepts those inputs with below error:
[2022/01/27 11:08:09] [ warn] [input:tail:tail.1] error scanning path: /var/log/nginx/*.log``` The issue is that the fluent-bit looks for logs on the node where the pod is running. On the node itself this folder exist: /var/log/containers Inside I have those for each pod: ```nginx-ingress-controller-xxx-pod.log -> /var/log/pods/default_nginx-ingress-controller-xxx/nginx-ingress-controller/0.log``` I can do sudo cat both and get logs fine. I will enable debug logs on fluent-bit to see
I restarted the pods a few times to check debug logs and started working…
At least I understand fluent-bit a little better
/var/log/containers folder is inside the node, not the pod? Makes sense i guess
You restarted fluent-bit and it just started to work?
I changed the log option to debug, then I started getting logs from nginx. then changed back to info and they are still working.
No idea, some bug maybe. Could be nginx, I need to update that too.
Maybe re-deploying fluent-bit did the trick