API Keys and Secrets: Where They Leak and How to Stop It
Most cloud breaches start with a credential that should never have existed. Here is where they come from, where they end up, and what to do instead.
Almost every cloud incident that becomes public follows the same shape: a long-lived credential existed, it ended up somewhere it should not have been, and someone found it. Not a clever exploit — a key in the wrong place.
The three kinds of credential
- User credentials — a password plus MFA. Protects console access, and the one people think about.
- Long-lived programmatic credentials — AWS access keys, GCP service account JSON, Azure client secrets, DigitalOcean and Linode API tokens. These work until explicitly revoked and they bypass the console entirely.
- Short-lived credentials — temporary tokens issued to a workload, valid for minutes or hours, rotated automatically. Almost nothing can go badly wrong with these.
The goal of everything below is to move as much as possible from the second category to the third.
Where they leak
| Route | How it happens | Prevention |
|---|---|---|
| Git history | Committed once, removed later — still in history | Pre-commit scanning; treat any committed key as burned |
| Public repository | A private repo made public with history intact | Scan before changing visibility |
| Container image layers | Copied in during build, deleted in a later layer | Multi-stage builds; never COPY secrets |
| CI logs | Echoed by a debug command | Mask secrets in CI; avoid set -x around them |
| Client-side code | Bundled into JavaScript shipped to browsers | Never put a cloud key in a front end |
| Shared documents | Pasted into a wiki or chat for convenience | Use a secret manager and share a reference |
| Backups and dumps | Configuration files inside an unencrypted archive | Encrypt backups; keep secrets out of config files |
Replace keys with identity
Every major platform can give a workload an identity instead of a stored key. Use it wherever it exists:
- AWS — IAM roles for EC2, ECS task roles, IRSA for EKS pods. The instance requests temporary credentials from the metadata service; nothing is stored.
- Azure — managed identities for VMs, App Service and Functions. Azure creates and rotates the identity.
- Google Cloud — attach a service account to the resource rather than downloading its key. Workload Identity does the same for GKE.
- GitHub Actions and GitLab CI — OIDC federation to AWS, Azure and GCP, so the pipeline exchanges a short-lived token for cloud credentials. No secret stored in CI at all.
OIDC federation for CI is the highest-value change most teams can make. It removes the single most commonly leaked credential class outright.
For the keys you genuinely still need
- Store them in a secret manager — AWS Secrets Manager or Parameter Store, Azure Key Vault, Google Secret Manager, or HashiCorp Vault. Not in environment files in a repository.
- Scope them narrowly. A key that can only read one bucket is a much smaller problem than one with account-wide rights.
- Set an expiry and rotate on a schedule you actually keep.
- Record what each key is for. Unattributable credentials never get rotated because nobody knows what will break.
- Alert on use from unexpected locations. Most platforms can surface this through their audit log.
What to do when one leaks
Speed matters more than tidiness. Automated scanners find public keys within minutes.
- Revoke the credential first. Not after investigating — first.
- Issue a replacement and deploy it.
- Read the audit log for everything that credential did, from its creation to its revocation.
- Check for resources you did not create, especially compute in unfamiliar regions, and new IAM users or keys.
- Check the billing page. Unexpected spend is the loudest signal of abuse.
- Rotate anything that credential could have read, including database passwords it had access to.
- Only then work out how it leaked, and close that route.
A key in a public repository is compromised the moment it lands there. Deleting the commit does not undo it.
Making leaks hard by default
- Add secret scanning to pre-commit hooks and to CI, so a key never reaches a remote branch.
- Add a
.gitignoreentry for.envand equivalents in every repository template. - Use short-lived credentials everywhere they are available, so a leak has a short blast window.
- Prefer resource-scoped policies over account-wide ones so the worst case is bounded.
- Review the credential inventory quarterly and delete anything nobody can explain.
Common questions
Is an environment variable safe enough?
It is better than a file in the repository and worse than a secret manager. Environment variables appear in process listings, crash dumps and some logging integrations. Use them as the delivery mechanism, with a secret manager as the source.
How often should keys rotate?
Every ninety days is a common standard, but the useful answer is “often enough that you have practised it”. A rotation process nobody has run is a rotation process that will fail.
Can I use one key across environments?
No. Separate credentials per environment mean a development leak cannot touch production, and revocation does not take everything down at once.
What about SSH keys?
Same principles. Use per-person keys rather than a shared one, use Ed25519, protect the private key with a passphrase, and remove keys when people leave.
Where to go next
Pair this with securing an account you did not open and least privilege without breaking everything.