Faster ways to fetch database credentials in Lambda

DevOps AWS
#1

At the moment, our lambdas fetch our database credentials using GetSecretValue()… this takes approx. 700-800 ms (this seems very slow to me, but I’ve no real frame of reference). Anyway, because this is so slow and thankfully most of our lambdas are provisioned, so we’ve been storing the credentials in a static Javascript variable (essentially a poor-man’s cache) which means that only the first lambda invocation wears the cost of hitting Secrets Manager.

However, we’re now implementing secret rotation which means my cache can become stale. I can see a couple of options (none of which are very appetising):

• Remove caching and every lambda invocation has a slow hit to Secrets Manager. Slow, plus a cost impact - all to cater for a once-in-a-30-day event.
• Add some sort of TTL to my cache. Better, but is really just reducing the window of the possible problem.
• Add some retry logic to Sequelize that recognises that the cached password is no longer valid, fetches the new password, saves in cache, and retries. Seems like the “best” option, but also seems like a lot of error-prone development and testing.
How are folks handling this particular problem? Is there a better way that I’ve missed?

#2

One more strategy could be to do something to reduce the fetching of secrets manager credentials down to something sensible. Not even sure if this is even possible?

#3

VPC lambdas can leverage Secrets Manager VPC endpoints, this might improve the latency, Lambda’s outside VPC’s, never really seemed slow. Now that you mentioned, might check how long it really takes (might provide you a reference)
Anywho, if it is 700-800ms for us, it seems to be acceptable, no ones complaining here.

#4

How about chucking it as a secret in SSM Parameter Store? Doesn’t solve auto secret rotation, might improve latency.

#5

https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/cache-secrets-using-aws-lambda-extensions.html

#6

https://developer.squareup.com/blog/using-aws-lambda-extensions-to-accelerate-aws-secrets-manager-access/

#7

Could be some useful stuff in there

#8

Effective secret rotation strategies always involve two versions of the secret active at the same time. We encrypt our secrets in source code, so after a secret update, we roll the encrypted text and deploy a new version. This ensures all new invocations are using the new one, then you deactivate the previous secret.

#9

Fwiw, we(square) never used that extension. it has the same expiration issue, its just caching on disk

#10

Good to know. :sweat_smile:

#11

Thanks for the thoughts folks. It sort of boggles my mind that the guidance Amazon offers is that setting up a web-server is the “best” approach. It just seems so brittle and doesn’t deal with rotation anyway :exploding_head:

Anyway, I’ve been doing a bit more research on how the alternating user rotation works. In the official documentation, I did find this line:
> If you use Alternating users rotation strategy, the credentials in the previous version of the secret are still valid and can be used to access the database or service
Double-checking the source code, it appears to me as though this is true… so, this should mean that even though I’ve rolled the credentials, the previous ones will still be valid (until I roll again).

As long as my local cache TTL is less than the rollover interval, I should be able to use the older credentials for a while

#12

Yep that’s what we do with our RDS secrets in SM. We rotate every 30 days, which means creds are valid for 60.