Calculate data traffic to/from Elasticache, S3, DynamoDB( no private endpoints )

DevOps AWS
1

Hello:wave: Any idea if is possible to get amount of data transferred to/from ( no private endpoints ) Elasticache, S3, DynamoDB, etc from VPC via public subnet? We want to interduce private subnets, natgw and private endpoints, but not sure how to calculate traffic ( single region, probably with two zones with zone affinity ). Thanks

2

Yes, you can do this with network flow logs.

https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs.html

Since you are sending to public interfaces, you would need to capture flow logs for everything and filter on destination. Then calculate from that.

3

Thanks. Even exporting flow logs to S3 and downloading them would cost something, right? What can I expect? Does it log each packet?

4

Do You have any suggestions on how to process it? Seems that Vector is capable to parse flow logs https://vector.dev/docs/reference/vrl/functions/#parse_aws_vpc_flow_log-examples-parse-aws-vpc-flow-log-default-format with eventual transform metric https://vector.dev/docs/reference/configuration/transforms/log_to_metric/.
But if there is another ready tool, I would rather use it, than making my own solution. Thanks

5

You can configure aggregation for the flow logs to the highest level that is supported for your instance types. For your use case, the higher the aggregation the better.

Check out athena, at least for doing simple queries. https://docs.aws.amazon.com/athena/latest/ug/vpc-flow-logs.html

6

Hi,
In billing, you could use cost and usage reports.

7

I don’t think you would get the data from cost until you switched to private endpoints though. Public endpoints in the same region are free.