How To Secure Application Credentials

You've written code that needs to make authenticated calls. How do you securely manage your credentials?

Authentication Primer

When authenticating, you attempt to prove you are who you claim to be. This is distinct from authorization, where you're granted permission to do certain things by whoever authenticated you. Usually these happen back to back, so seeing a distinction can sometimes be a bit blurry. Many times the existence of your identity in an authorization look up simultaneously authenticates you. There are plenty of other schemes though that do have a distinction where one party attests to your identity while another uses your identity to assign privileges.

For a physical example, when you make a reservation, you provide your name as a form of shared secret. When you arrive, they first ask for your name to authenticate you made a reservation. Then they take you to the table you're authorized to use, letting you skip the line. For a tech example, this is how web session cookies work. Your authentication code or single sign-on provider attests to the identity of a user based on their credentials. Once authenticated, a cookie's set in the user's agent so following requests are tagged with the identity assertion. From then on, all your other code can authorize privileges and private resources based on the authenticated identity.

When considering authentication protocols, there are generally 3 models:

1. Something You Know
2. Something You Have
3. Something You Are

Authenticating to third parties via APIs is usually restricted to something you know (a token, password, cryptographic keys, etc.). In this model, it's imperative you do not allow knowledge of the secret to be shared beyond your trust boundary. When using something you know, it's possible to copy the means of authenticating indefinitely. If control of the secret is lost, so is the means of restricting and meaningfully logging access. The only recovery is to change the shared secret and more appropriately share and secure it going forward.

Threat Models & Trust Boundaries

A trust boundary is important when creating a threat model. Your threat model defines the types of attacks you plan to consider. Without a threat model, you're just changing things. Good security relies on defining the threat(s) your proposed change(s) mitigate. For example, you may decide your threat model assumes the users of your system may be malicious (or not). You may decide that not even you should be able to read users' data (or not). You may decide you will not try to secure against malware on a machine but instead secure against it getting there in the first place (or vise-versa).

All of these dramatically change the form and capabilities of the system you build. They also all come with costs involved from engineering time and effort, to user training and support, and just plain old system complexity. This is often why attempting to make a product secure at the end of its development is not a smart decision. If you're design relies on insecurity you now wish to defend against, it can actually sometimes be impossible to secure without a considerable redesign. Doing so right at the end before launch is orders of magnitude more work than continuous consideration throughout the project.

A trust boundary in your threat model defines the interface(s) between those trusted to act in the interest of the system, and those who may be malicious. Few if any of those in an untrusted group will be malicious, but assuming malice encourages us to verify before trusting. Why things are the way they are is beyond the scope of this post, but can be summarized as, humans suck. As a result, the smaller your trusted group and more accountable those inside are, the better your security.

However, this often comes at the cost of complexity and flexibility. Security is always about reducing ease of use to prevent incident likelihood and remediation costs. The best security designs are, in my opinion, near invisible to those acting in good faith. Always remember, just trusting the operator of a system isn't enough. You also have to trust every line of code that runs on the system(s) they use. For example, your users may not attack you, but someone subjugating their system well may.

Now let's examine some of our possible solutions to the problem we started with: management of third party credentials.

Credential In Source

When writing code, there are a number of methods for managing credentials. The first is to put them in the source code and/or code repository. This method is by far the easiest and most flexible, however, it relies on secrecy of the code itself for secrecy of the credentials. Bold statement, but this is almost always a bad idea. Unless you plan on only ever writing, storing, compiling, and running this code on a single system, you can almost guarantee it'll be copied all over the place without thinking about the security implications. At some point your code (and now secret credentials) will be available to someone who really shouldn't have access.

For example, the moment you use a centralized repository server, anyone with access to that system, by administrative privilege or software exploit, could potentially obtain your credentials. As another attack, build servers (all the rage thanks to CI/CD) are notoriously insecure as they have the challenge of isolating execution despite preserving artifacts, all while supporting hundreds of systems, services, and tools used by developers.

Remember that at some point, something will have to be trusted. You can restrict all you like, but you cannot change the fact that some piece of code or hardware will need dangerous privileges. You can only reduce your attack surface so much. To overcome this, the best strategy is usually one that layers defences, forcing an attacker to overcome multiple hurdles in order to be successful. Finding a weakness in just one layer is now no longer enough, and the difficulty becomes the multiplied challenge of overcoming each layer. To add a layer to this method, you can generally do one of two things.

Credential Service

One popular method is to layer something you have on something you know. An example of this is to create or deploy a service that requires proof of ownership (something you have). A common approach is using an IP address allow list and TCP's three way handshake as a proof of ownership protocol. This relies on the security and stability of network routing and address assignment to secure authentication. In this model, a single trusted third party service could contain all of the secrets. Then, each of our services could contact this service to either obtain its credentials or better still, have this third party authenticate or act on its behalf.

The biggest problems with this method are on its assumption of security critical trust in the network and the new single point of failure. For example, it assumes you can know what IP addresses your services will be assigned and that an attacker cannot arrange to also convince your credential service that it possesses those. The use of static addresses are becoming generally frowned upon in an IPv6 world. Use of static addresses also makes denial of service (DoS) mitigations harder. Additionally, popular services like App Engine, Lambdas, Cloud Workers, and other serverless vendors either do not allow static addresses or make using them problematic or insecure. If you cannot trust your infrastructure treats IP addresses as security critical, you cannot trust this method for authentication. You can add DNS as an abstraction over IP addresses, but now you also have to trust the unauthenticated unencrypted poisonable DNS infrastructure in your security critical infrastructure. Many other systems do rely on DNS security (like TLS thanks to ACME), but it is still an extra point of attack in this model.

One way to compensate is requiring credentials for your credential service. This has merits, but all the same problems putting credentials in your source did in the first place. Although, you've now just introduced a single point of failure. To understand the single point of failure, imagine what happens if that service is unavailable. Also, if your attacker has an exploit for this system, they may now have all of the credentials to all of your services. For a bit of history, the NSA documents showed that well funded attackers hunt SysAdmins.

The biggest upside to this however is key rotation. No service knows the credentials or has them hard coded when done right. You can then generate and use new credentials automatically and continuously. I'm not a proponent of rotating keys all the time unless they've been compromised, but for those that are, this is often a compelling feature. Compromise by the way can just include when someone who was trusted should no longer be (retirement, termination, resignation, etc.).

Many people go this route as an attempt at isolating developers from production. I'll just say right now, if you can't trust your developers, hire ones you can. If I can run code on your machine, I can do just about anything I want.

Credential Encryption

Another method is to encrypt the credentials while they reside outside of your trust boundary but enable those trusted to decrypt them when needed. Two popular methods for doing this include git-crypt and git-secret. These are convenient because they look and feel almost identical to just putting credentials in the source code, but transparently encrypt them whenever they sit on an untrusted computer. They also allow you to open source your code without providing access to the secrets.

These work by encrypting and decrypting something from STDIN to STDOUT, applied using Git Attributes. Specifically, the clean and smudge filter attributes. Using the clean filter, a file can be modified transparently before it is added to git's index (and subsequently committed). Using a smudge filter, blobs in git can be transparently modified before being written into your working directory. The subtlety here is in key management. The simple answer is use PGP, but a deeper discussion of this solution is set for a future post.

With this solution, development doesn't always require credential access. People using encrypted versions can still use and test the software without the functionality provided by the secrets. You can also allow users to drop in their own credentials. This means you can still have SysAdmin only and developer only credentials if you want better production isolation.

While PGP has been around for what feels like forever, many people still find it hard to use and unintuitive. Thankfully it's essentially set and forget. The only complex time is when you have to create your keypair and when you create a new repo. When you're doing routine commit cycles it can be so invisible you forget it even exists from time to time.

Make sure you don't accidentally commit the credentials in the clear. This mistake is usually only made at the start of a repo, but can happen when you overwrite credentials in a locked repo. If you get them into your history, getting them out is possible but a pain as you'll have to rewrite the history. I often advise just rotating the keys when this happens to destroy the old credentials.

On the topic of key rotation, this method does nothing to help. You also don't gain any additional monitoring. It doesn't provide a backup in case you forget to backup your PGP keys or remember your password, nor can you see the credentials in any sort of website based git view. You've also added another tool to your build chain that developers will have to install, configure, update, and learn.

Conclusion

Don't think of any as your best choice. There are a large number of considerations that make each a good choice given a different situation. You have to think carefully about your trade offs in any engineering decision.

I hope this gave you a good introduction into threat modeling, authentication schemes, security fundamentals, and making trade offs. If you're just putting security credentials in your code, I strongly encourage you to think critically about the long term implications that has.