Implementing Secure and Scalable Secrets Management for Multi-tenant Environments

DevOps Kubernetes
#1

Hey guys, wanted to get your take on best practice to manage secrets management securely and operationally easy. We currently have a b2b product where…

  1. each client is single tenant (has a set of ~100 configuration variables for their environment)
  2. we use 1password and keyvault to store secrets, and have a local script to update keyvault with the latest config on 1pass
  3. helm charts, Kubernetes CSI secrets store (for KV → secrets pull per pod spin up)
  4. with more clients, updating secrets globally across all clients is getting more untenanble
    Are there secure solutions out there that would make our lives easier? (i.e. secrets update across all client secrets client1/secret1 , client2/secret1 without having to propagate it everywhere manually). Writing more scripts could work, but we’d prefer to learn the best practice. Preferably within Azure / Hashicorp land to keep our SBOM small.
#2

We use HCP Vault to store our secrets, a very small plan that doesn’t cost us much. Then, you use the Vault provider in your Terraform, or plugin for CD (Jenkins for example). Works great. We use 1Password too, but for human users as a local password management tool on their laptops, not programatic usage for non human consumption.

#3

oh, and we pull our kube secrets from vault too… forgot that part lol. My mind was on terraform.

#4

Hey - thanks for responding. I’ve thought of Vault + Terraform, here’s some questions

  1. does the terraform state file store the passwords in plain text if vault is used to pull secrets in? Ideally we have key management (i.e. if key gets added) baked into terraform as well, but we prefer not to expose the secrets on plain text if vault acts as a data source
  2. Has vault been a good way to update your config across many clients? (we’re on single tenancy setup and managing these repeated secrets per client has been fairly painful) - preferably there would be a way to update repeated secrets across clients in a simple manner
#5
  1. I believe even if you set the secret as sensitive in terraform, it will still store the secret(s) in plain text… depending on what route you take. I don’t have Terraform Cloud with HCP, however I do have our state S3 bucket using encryption at rest, so unless someone has access to decrypt… I feel its safe that way. Here is some more on that; https://developer.hashicorp.com/terraform/tutorials/configuration-language/sensitive-variables#sensitive-values-in-state
  2. I work for a small company, and using vault has been very nice. We have our secrets per environment / cluster / namespace in Vault and its very easy to update. You can update via web GUI or vault cli. We only have 3 EKS clusters (and 1 sandbox), and I feel its very easy to use in general

I know folks like us don’t love talking to sales people, but I would hit up Hashicorp and ask for a quick meeting.