Displaying location data from Loki logs in GeoMap panel

Monitoring and Observability Grafana
1

Hello,
i send netflow data from my network devices to loki as JSON log lines. the logs contain latitude, longitude, country code, city, state. however using loki as datasource in a geomap I am not able to select the JSON parsed values as “labels” and I am not able to configure the panel. I tried “line_format” in LogQL to get only some parameters in the log line - however I have no success.

so the main questions I have are:

  1. is it possible to use the GeoMap Panel to display the location based on loki datasource log lines in general?
  2. can someone help me to find the correct format or how to transform the log data to visualize the locations properly?
    Note:
    Adding the lat/lon as labels at ingest time is not a good idea because of high cardinality. maybe I could use country codes as a label - however I would like to avoid this if possible.
    As I do not need the location 100% accurate based on lat/lon. I think country codes is enough.
2

I do not use alloy to send the logs to loki so I can not use alloy geolocation stages. I use this software to collect the netflow logs, add the geolocation information and send it as JSON to loki:

https://github.com/netobserv/flowlogs-pipeline

3

Do you need lat/lon or just country code ? The Geomap supports both but obviously country code is easier because it’s one field only with a simple format

I would recommend to add the enrichment as structured metadata: no need to parse then and no indexing done either.

4

In the geomap Panel i have to select the lat/Lon fields in my Data. But the Loki Log only Provides “Line” as a Field so no possibility to select lat and Lon.

So i have all Data available but only as Log Line and Panel does Not allow me to select a Part of the Line’s content as lat or Lon.

Does this make Sense what i try to explain?
Maybe i Miss Something.

5

Check <https://play.grafana.org/d/T512JVH7z/loki-nginx-service-mesh-json-version?orgId=1&from=now-6h&to=now&timezone=utc&var-datasource=ac4000ca-1959-45f5-aa45-2bd0898f7026&var-label_name=filename&var-label_value=%2Fvar%2Flog%2Fnginx%2Fjson_access.log&var-job=$__all&var-instance=$__all&editPanel=14|this example>

6

It’s not exactly the same format, for sure

7

But you will get the idea (using country code, not lat/lon)

8

Hard to understand all using my small mobile Phone Screen, but it Looks Like i missed the “sum by()” part. If i First Format the Line and then szm by country Code or lat/Lon maybe the Panel understands it.

Will try this later this day. Good hint.
Thank you!

9

I got it working.

I first parsed the log line with | json and extracted the latitude and longitude fields. within the json parsing I renamed the field from its original naming DstLocation_longitude and DstLocation_Latitude to lon and lat .

then I did a count_over_time(...[$__auto]) to count the amount of these logs

and the last step was to do a sum by (lon,lat) (....) and the Panel could autodetect the lon/lat values and place the marker correctly.

Later I extracted a few more json fields like country, region, city, dstAddr and added this to the sum by () to make it available in the geolocation tooltip.