Enhancing Grafana Loki for searching API server logs: grouping, table view, permalink, and field suggestions

Monitoring and Observability Grafana
#1

Hi ! After evaluating a lot of different options (OpenSearch / Google Cloud Logging / Amazon CloudWatch / NewRelic Logs…), we are considering using Grafana Loki to view our logs.
From our evaluation, Grafana Loki itself seems truly promising but we feel there are some shortcomings regarding the Grafana Explore GUI. We are wondering how everyone is dealing with the following use-cases (mainly for searching API server logs) ?
• Group all logs related to the same request
◦ You usually want to group all logs by request-id/trace-id so that only one representative log is visible per request at first, and then you can see details of request logs by clicking on it
◦ It is of course possible to filter logs by some request-id/trace-id, but it requires a few more steps which makes it difficult to quickly switch to a different request.
◦ Also, there is quite a lot of noise making it difficult to grasp for example how many requests are returning 500 errors etc.
• Table view of specific fields
◦ Sometimes you want to pickup a few fields from a json log and display them as a table (like: time | API name | status code | duration | message )
◦ Currently, you can click on the “eye” icon in the “Detected fields” to view specific fields, but :
:black_small_square:︎ Only detected fields are supported. When you json parse Loki logs, almost all fields become log labels and thus you cannot select them
:black_small_square:︎ The display format is field1=value1 field2=value2 (logfmt), which is not as readable as a table, where column widths are adjusted and values padded
◦ It is possible to play with line_format and tabs/padding but this is quite time-consuming
◦ It is possible to create a Grafana Dashboard with a transform operation, but the overhead is huge
• Share a permalink to a specific log
◦ For example in Google Log Explorer you can obtain a permalink to a specific log just by clicking on it. It is quite convenient to share a log to a teammate via Slack etc.
• Have some field search suggestions
◦ When you don’t know in advance what are the different fields contained in some service’s logs, it can be difficult to write a search query. Loki labels have suggestions which makes it easy to filter by labels, but usually you also want to easily filter by log fields.
◦ As Loki doesn’t index log data, it is difficult to give suggestions before looking up logs, but it would seem possible for Grafana to compute some suggestions based on the logs retrieved after an initial lookup. Something like that:
:black_small_square:︎ You first look for log data just by filtering via labels
:black_small_square:︎ Grafana gets 1000 log lines from loki, and analyzes them to compute search suggestions
:black_small_square:︎ You can have some field search suggestions for the next query
Would anyone have some insights ? Thanks in advance !

Other than that, Grafana and Loki are truly amazing and we would love starting using them! :smile:

#2

Hello, thanks for the research and the write up :smile: I sent this over to the logs team to take a look at. For the share a permalink to a specific log, they pointed me to this feature discussion which you can contribute to and/or upvote https://github.com/grafana/grafana/discussions/37621

#3

> • Group all logs related to the same request
When you explore a tempo trace, you can click to show all logs related to that trace/span.
> • Table view of specific fields
Hmm can’t you just click to view your data as a table? I’m attaching an image to illustrate what I mean.
> • Share a permalink to a specific log
That would be really cool. There’s a new snapshot feature that you could use to share a specific state of a dashboard, but that’s a different thing.
> • Have some field search suggestions
That would be cool too. There’s the favorite query feature etc that your team could use to help remembering fields, but that’s a different thing.

#4

Thank you so much for your reply!

> For the share a permalink to a specific log, they pointed me to this feature discussion which you can contribute to and/or upvote https://github.com/grafana/grafana/discussions/37621
I will definitely check it out, thanks!

> When you explore a tempo trace, you can click to show all logs related to that trace/span.
We also tested Tempo, which is great by the way ! :+1:
As you pointed out, we can show all logs related to a certain trace/span from Tempo, and we can also show all logs related to a specific request/trace by filtering by request-id/trace-id In Grafana Explore.
I guess my observation is more about showing only one representative log line for each request so that you can have a general overview of what’s going on, and then click on a request to show all logs of that request to have more details.
I attached some screenshots of a Google Log Explorer experience, to give a better idea. In the first one, you can see one log line per request, giving you a clear overview of which APIs are failing. Then by clicking on a request, you expand it to show more details.
In Loki explore, as there is no grouping all log lines are evenly displayed:
• log lines corresponding to different requests are mixed together, making it difficult to read
• you cannot really grasp which APIs are impacted by an issue or what’s going on as there is a lot of noise due to detailed log lines
Just an idea: the same way you can define derived fields in loki data source to switch from a log to a Tempo trace, it could be interesting to be able to define some grouping fields, so that logs are grouped by defaults by those fields ? Of course this is easier said than done :bow:

> Hmm can’t you just click to view your data as a table? I’m attaching an image to illustrate what I mean.
You certainly can view logs as a table, but in my understanding this requires creating a panel/dashboard.
This is nice when you want to show a specific view of your logs (like certain fields) and you have time to prepare in advance, but this does not seem suited to a situation where you want to constantly change which fields you are viewing in order to investigate/debug an issue. I added some screenshots.
To create a nice table, you need to :
• create a table panel
• write a complicated query like the following
```{container=“app”, app=“my-app”} | json | line_format “app={{.app}} req_id={{trunc 8 .request_id | trim}} level={{.level}} type={{.in_request_type}} service=’{{.in_request_service}}’ method=’{{.in_request_method}}’ ua=’{{trunc 10 .in_request_ctx_ua}}’ {{ if .in_request_status_code}} status_code={{.in_request_status_code}} req_duration={{.in_request_duration_ms}}ms {{- end}} msg=’{{.msg}}’ {{ if eq .out_request_type “sql” }}sql=’{{.out_request_body | replace " " " " | replace " " " " }}’ {{- else if .out_request_type }} out_type=’{{.out_request_type}}’ out_service=’{{.out_request_service}}’ out_method=’{{.out_request_method}}’ {{- end }}” | error=`````
• add an extract fields Transform
• add an Organize fields Transform
The first time I created such a table it took me like 30minutes to achieve this result. I am sure I can do it more quickly now but even so I wouldn’t use it on live investigations.

In Grafana Explore (Loki datasource), you can pickup some fields to be displayed, but :
• when you json parse your logs some fields are not selectable (see screenshot)
• fields that you can select are displayed, but in logfmt format which is not as readable as a table
• by writing line_format you can adjust the display, but you need to play with tabs and padding to get a nice view, which takes time
Being able to also select log labels, and to have the choice to display selected fields in a table would greatly improve the search experience (first you look at raw json logs, then pickup some fields to be displayed so that you can correlate log lines affected by an issue, then quickly switch to other fields to narrow down your search etc.)

> Share a permalink to a specific log
>> That would be really cool. There’s a new snapshot feature that you could use to share a specific state of a dashboard, but that’s a different thing.
Thank you for sharing the info. There seem to have some workarounds so I don’t think this is a big issue, it just would be nice to have!

> Have some field search suggestions
>> That would be cool too. There’s the favorite query feature etc that your team could use to help remembering fields, but that’s a different thing.
Thanks! I will check out if we can come up with some workarounds using favorite query feature!

Sorry for the long post. My feedback may seem a little bit negative but as I pointed out before I evaluated a lot of systems and I can say with confidence that apart from the points above Grafana Loki is by far the best solution right now, especially as an Observability platform (easy switch from/to metrics,log,traces (profiling?), low ingestion cost, very stable, nice dashboards, lots of plugins, lots of activity…).
By the way I watched almost all your webinars, they are amazing! Keep up the good work :slightly_smiling_face: