Hi ! After evaluating a lot of different options (OpenSearch / Google Cloud Logging / Amazon CloudWatch / NewRelic Logs…), we are considering using Grafana Loki to view our logs.
From our evaluation, Grafana Loki itself seems truly promising but we feel there are some shortcomings regarding the Grafana Explore GUI. We are wondering how everyone is dealing with the following use-cases (mainly for searching API server logs) ?
• Group all logs related to the same request
◦ You usually want to group all logs by request-id/trace-id so that only one representative log is visible per request at first, and then you can see details of request logs by clicking on it
◦ It is of course possible to filter logs by some request-id/trace-id, but it requires a few more steps which makes it difficult to quickly switch to a different request.
◦ Also, there is quite a lot of noise making it difficult to grasp for example how many requests are returning 500 errors etc.
• Table view of specific fields
◦ Sometimes you want to pickup a few fields from a json log and display them as a table (like: time | API name | status code | duration | message )
◦ Currently, you can click on the “eye” icon in the “Detected fields” to view specific fields, but :
︎ Only detected fields are supported. When you json parse Loki logs, almost all fields become log labels and thus you cannot select them
︎ The display format is
field1=value1 field2=value2 (logfmt), which is not as readable as a table, where column widths are adjusted and values padded
◦ It is possible to play with line_format and tabs/padding but this is quite time-consuming
◦ It is possible to create a Grafana Dashboard with a transform operation, but the overhead is huge
• Share a permalink to a specific log
◦ For example in Google Log Explorer you can obtain a permalink to a specific log just by clicking on it. It is quite convenient to share a log to a teammate via Slack etc.
• Have some field search suggestions
◦ When you don’t know in advance what are the different fields contained in some service’s logs, it can be difficult to write a search query. Loki labels have suggestions which makes it easy to filter by labels, but usually you also want to easily filter by log fields.
◦ As Loki doesn’t index log data, it is difficult to give suggestions before looking up logs, but it would seem possible for Grafana to compute some suggestions based on the logs retrieved after an initial lookup. Something like that:
︎ You first look for log data just by filtering via labels
︎ Grafana gets 1000 log lines from loki, and analyzes them to compute search suggestions
︎ You can have some field search suggestions for the next query
Would anyone have some insights ? Thanks in advance !
Other than that, Grafana and Loki are truly amazing and we would love starting using them!