What to do if you accidentally committed secrets to a repository

What should you do if a team member (side note: It might have been the intern who simply didn’t know better, or may have been the chronically overworked senior developer. It may even have been you…) has committed a secret like an API key, a database password, or a JWT secret to a Git repository? Regardless of whether the commit was intentional or accidental, in this post I’ll briefly discuss some measures that you should take now to prevent unauthorised access to accounts and systems, as well as steps you can take to prevent this from happening again in the future.

Once a secret has been pushed to a repository that is shared with others, it’s no longer secret. The severity of this depends on at least two things:

• What is the secret used for, and what happens if bad people (side note: The technical term for this is “malicious actors”.) get their hands on it? If you leaked the admin password of your app or server, you’re in big trouble and need to act now. On the other hand, if the leaked secret is an API token that’s only used in one specific test environment, there’s probably no need to panic.

• Who shouldn’t have access to the secret but now do? If you pushed the secret to a public repository, that’s effectively everyone. If it was pushed to an internally hosted private repository, it might be no one. And if it’s a private repository on a cloud provider… well, that depends on the amount of trust you place on them.

Ideally, you should never reuse secrets. Not between systems, not between environments, and certainly not once someone has accidentally exposed them. This is why your first step should be to change or rotate the leaked secret (side note: Only skip this step if you are very sure that the impact of the leak is basically zero!).

Depending on how your apps are configured, this may cause a temporary outage. While that’s not ideal, it’s still far better to break things yourself than to have someone else do it – especially if they also steal your data, wipe your databases, or worse!

If, for whatever reason, you have decided that you want to continue using a leaked secret after all, or want to prevent more people from discovering your embarrassing mistake, you can try to remove the committed secret from the repository’s Git history.

Depending on how long ago the secret was committed, there are two approaches you can use. Note that both methods will require you to be able to force-push to the branch, which in most repositories is no longer possible once the commit has been merged into the main branch.

If the secret was committed very recently, the easiest way to remove the secret from Git history is by recreating the last commit(s). First, git switch to the branch where the secret was committed, e.g. your pull request branch or the main branch.

Then, look up when the secret was committed. If the secret was added as part of the last commit, run the following command to reset the Git history to the point right before the commit was created:

The 1 here tells Git how many commits you want to undo. For example, if the secret was added three commits ago, you would update the command as follows:

After running the git reset command you should see that the Git history has returned to an earlier point in time, but the changes are still present in your working directory.

You can now recreate the commits manually without the offending secret, using a combination of git add and git commit (side note: Or simply use a visual Git client, that’s what I do.). When you have made your changes, use git push --force to overwrite the branch on the remote:

The method above works well if the leak was discovered quickly, but what if the secret was committed a very long time ago?

The good news is that you can still purge it from your Git history. The bad news is that it doesn’t make a lot of sense, because y’know, the leaked secret has been out there for so long it’s kind of become your own version of 09 F9 – it’s basically public information at this point. On top of that, it involves a very, very, very nasty procedure that is going to make a lot of people very angry.

To remove secrets from commits that were made a long time ago, you can use git-filter-repo. I’m not going to tell you how to use it because you shouldn’t actually be doing this, but if you do, make sure to back up your repository first!

How do you find out whether someone has committed secrets to a repository? You can of course browse through the repository’s files and pull requests in the hopes of catching a leaked secret, but using a tool like Gitleaks is much easier.

Gitleaks is an open-source command-line tool that helps you find secrets like passwords, API keys, and tokens in your repositories. Gitleaks binaries for Linux care available on the releases page, while macOS users can install Gitleaks via the Homebrew package manager:

If you don’t want to go through the hassle of installing Gitleaks on your machine, you can also run it directly via Docker:

The first five lines here are boilerplate that set up the Gitleaks binary in a container and ensure it has access to your current directory. The last line specifies the command-line arguments that are passed to gitleaks. In this case, the arguments instruct it to scan the current directory for plain-text secrets and show you which secrets it has detected (--verbose).

If you don’t want to type that out every time – I know I wouldn’t – learn how to make better use of your shell history, or append the following line to the end of your .bashrc or .zshrc file to create an alias for Gitleaks:

This will allow you to invoke the containerised version of Gitleaks as if it is a local executable:

Note that the first time you run Gitleaks, it may detect secrets that have already been committed a long time ago. If you want to ignore these for now so that you can focus on preventing new secrets from being committed to the repository, you can add a baseline that works similarly to a .gitignore, but for secrets.

Although you can run Gitleaks manually to detect secrets that are already present in a repository, what you really want is to prevent them from ending up in there at all.

There are two common ways to do this: Git hooks and CI pipelines.

The best method by far is to ensure that secrets never make it into commits in the first place. This can be done by installing a pre-commit hook that tells Git to automatically run Gitleaks on your changes right before creating commits. If the Gitleaks check passes, the commit will be created – if not, the commit will be aborted.

While Git hooks are a good first line of defence, they are not something you can always rely on as not all contributors may have installed them.

This is why you should also include a check in your continuous integration (CI) pipeline that verifies that code submitted in pull requests (side note: All changes have to made via pull requests and no one commits directly to main, right? Right?!)does not contain any secrets.

The snippet below shows an example of a GitHub Actions workflow that runs Gitleaks for pushes to pull request branches and the main branch:

By now, it should be clear that you should not commit secrets into your repository, but you still may need a way to share secrets. There are ways to do this safely, without committing plain-text secrets into your repositories.

You can, for example, use a password manager with business features like 1Password or Bitwarden. A major benefit of such password managers over those of Apple and Google is that they allow you to easily share passwords with others in your team. Moreover, they allow you to save not only passwords, but also other types of secrets, like tokens, certificates, and authenticator codes.

Password managers are great for secrets that you can provide interactively, like passwords. But if you want to do automated deployments, you will need a solution that can be used without manual intervention. Consider using environment variables – or better yet, environment secrets – if you want to store secrets on servers or in deployment tools. This requires you to trust the server or tool where the secrets are kept.

If you don’t (want to) trust any third-party servers or tools or if you still want to store secrets in your repository because you feel it’s more convenient, you may want to consider encrypting them using a tool like SOPS. This allows you to store secrets safely along with your code.