How to Rotate Your API Keys After a Security Breach (Step-by-Step Guide)
Introduction
When a security incident compromises your cloud environment, swift and decisive action is critical. Recently, Braintrust—a startup providing an operating system for engineers building AI software—confirmed that hackers breached one of its Amazon cloud environments. As a result, every Braintrust customer has been advised to rotate their sensitive API keys. This guide walks you through the exact process of rotating those keys, ensuring you maintain the security of your AI applications and prevent unauthorized access. Whether you use Braintrust or another platform, these steps apply to any scenario where API keys may be exposed.
What You Need
Before you start, gather the following:
- Admin access to the cloud console or service where your API keys are managed (e.g., AWS, Azure, GCP, or the third-party platform itself).
- List of all applications or services that use the affected API keys. This includes internal tools, production systems, development environments, and any integrations.
- A secure vault or password manager to store the new keys temporarily.
- Access to CI/CD pipelines or deployment scripts if keys are embedded in code repositories or config files.
- Communication channels (Slack, email, ticketing system) to coordinate with your team and minimize downtime.
- Logging and monitoring tools to track API usage and detect anomalies after rotation.
Step-by-Step Rotating Your API Keys
Step 1: Confirm the scope of the breach
Before touching anything, verify exactly which keys are compromised. Check any official notifications from the service provider (e.g., Braintrust’s email). Look at your usage logs for unauthorized access attempts. Identify the key IDs or names that were in the compromised environment. If the provider has issued a list of exposed keys, use that as your starting point. Otherwise, assume all keys associated with that account are at risk.
Step 2: Generate new API keys
Log into the relevant cloud or service dashboard. Create new API keys for each affected resource. For example, in AWS IAM you would create new access keys; on the Braintrust platform you would generate fresh API tokens from the settings page. Do not delete the old keys yet. Save the new keys in a secure location, ideally directly into your secrets manager (like AWS Secrets Manager or HashiCorp Vault).
Step 3: Update all code and configurations
Replace every occurrence of the old API key with the new one. This includes:
- Environment variables in your application servers
- Configuration files (e.g.,
.env,config.yml) - CI/CD secrets (GitHub Actions, GitLab CI, Jenkins)
- Hardcoded strings in source code (run a grep search across your repositories)
- Third-party integrations that use the key (e.g., monitoring tools, chatbots, data pipelines)
Tip: Use a search-and-replace tool or a script (like sed or aws ssm put-parameter) to automate this and reduce human error.
Step 4: Deploy the updated keys to production
Once all files are updated, deploy the changes to your live environment. For many teams, this means a rolling deployment or a restart of services. During this step, ensure you have a rollback plan in case the new keys cause authentication failures. It’s wise to test in a staging environment first. After deployment, verify that your applications can connect and function normally with the new keys. Check API response codes and error logs.
Step 5: Revoke the old API keys
After confirming the new keys work everywhere, go back to the service dashboard and disable or delete the old keys. This step prevents attackers from using the compromised credentials even if they still have them. On AWS, you can deactivate a key first and monitor for any lingering failures before deleting it entirely. On Braintrust, look for an option to revoke the old token. Do not skip this step.
Step 6: Rotate any related secrets
Often, API keys are not the only secrets in a compromised environment. If the breach also exposed access tokens, database passwords, or SSH keys, rotate those as well. Review the security advisory from the provider—they may indicate a broader set of credentials to rotate. For Braintrust, only API keys were mentioned, but it’s prudent to audit all secrets stored in the same location.
Step 7: Monitor for unusual activity
After rotation, increase your monitoring. Check logs for any authentication attempts using the old keys—this tells you if attackers are still trying. Also watch for anomalies like unexpected data access (in Braintrust’s case, AI model evaluations or stored prompts). Set up alerts for failed authentication and unusual API call patterns. Continue monitoring for at least 72 hours.
Step 8: Communicate with stakeholders
Notify all relevant teams—engineering, security, support, and your customers if required. In the Braintrust situation, every customer was directly contacted. Internally, share a brief incident report explaining what happened, what steps were taken, and any changes to workflows. This builds trust and helps your team react faster in the future.
Tips for Future Prevention
- Use short-lived keys whenever possible. Services like AWS now support temporary credentials (STS) that expire within hours.
- Rotate keys regularly, even without a breach. A quarterly rotation policy reduces the window of exposure.
- Centralize secret management with a tool like AWS Secrets Manager, Azure Key Vault, or HashiCorp Vault. This makes rotation a single operation.
- Implement least-privilege access for your API keys. Only grant the minimum actions needed (e.g., read-only for evaluation if that’s all Braintrust requires).
- Audit your cloud environment after a breach. Check for misconfigurations like public S3 buckets or open IAM policies that may have contributed.
- Enable multi-factor authentication (MFA) for all console users. Even if keys are stolen, the attacker cannot login to the dashboard.
By following these steps, you can confidently respond to an API key exposure like the one Braintrust experienced. The key is to act fast but methodically—never try to rotate everything at once without testing. Security is a continuous process, and this guide gives you a repeatable plan for one of the most common post-breach actions. Remember to document everything for your incident response playbook.