Duplicate API logs in Datadog

DevOps
#1

Any idea on how to stop duplicating API logs in datadog. Currently request to my api are logged twice.

#2

The subscription field is 2 different values

#3

What do you mean with respect to subscription field

#4

The solution is to step sending the log twice most likely :stuck_out_tongue:

#5

I mean it has 2 logs for one request

#6

Is the app emitting logs twice? Are there two datadog agents collecting them independently? Are you collecting logs from two different proxies that look identical?

#7

Yea, you’re likely somehow producing the log once, and you’ve set things up such that you’re consuming it twice

#8

Consuming in-container app logs that also get mirrored to STDOUT, as well as consuming STDOUT is one common scenario

#9

Each log event should, I think, have some form of provenance info on it

#10

I.e. which agent(s) sent/emitted the event

#11

And via which channels

#12

Is the app emitting logs twice? Yes
Are there two datadog agents collecting them independently? I can see 2 subscription values
subscription projects/aws-account-id/subscriptions/datadog
subscription projects/aws-account-id/subscriptions/loadbalancer-datadog

#13

Are those like cloudwatch events or something?

#14

Can you share an actual instance of the event, with all/most of the metadata fields?

#15

You should hopefully have some understanding of where you’re gathering logs from

#16

One way or another, if you don’t want duplicate log events, you’ll either need to figure out a way to filter in the log pipelines, unsubscribe from a particular event stream/channel that has duplicates, or tell datadog agents to stop consuming events from something

#17

There should be a shitload of tags on the log events

#18

Container_name, pod_name if in k8s

#19

Basically, DataDog will do what you tell it

#20

If you tell it to listen to something, it’s going to listen to it and send the events