Last week Twitch faced the kind of nightmare scenario that every online service hopes to avoid: their source code and database dumps were leaked on the internet and broadly distributed. Database dumps exposed details about their business, everything from user complaints to streamer payouts. And the source code uncovered the inner workings of their service and infrastructure—identifying privileged VPN users, key hostnames, and all application components, as well as all the passwords and keys needed to access those resources.
Getting to the bottom of breaches like this requires thorough investigation, outside audits, and patience. None of us should expect quick answers, but that hasn’t stopped a lot of outsiders from speculating. Regardless of what the final causes are, there are some lessons we can draw from this.
Source code is both an asset and attack vector
The corollary to “every company is a software company” is that every company’s software is a strategic asset, and the source code reveals the inner workings of the company.
And as the scope of what we manage with code—the infrastructure as code trend has many companies increasingly using gitops to manage every service they interact with—the significance of code as an attack vector only increases.
It’s not a secret if you write it down
The old definition of a secret was something you only told one person at a time. Putting secrets in a repo is an expedient solution to the problem of managing code and resources that require secrets, but while code is often meant to be shared, secrets aren’t.
Developers often make repos public (sometimes unintentionally) just to solve a problem, code moves from repo to repo, and the history in our repos reveals everything.
Tools now exist to securely manage secrets. I’m a fan of the portable and open-source Hashicorp Vault, but every major cloud has a solution as well. Unfortunately, these tools are still new, and many organizations have complex systems built around insecure secret storage.
Balancing the access developers need to do their jobs against the policy of least privilege is hard
We invite people into our repos because collaboration leads to innovation, and nothing kills motivation and progress faster than a developer waiting on access to a repo. High-performing developers—ever cautious of Conway’s Law, and eager to innovate across the organization—bristle at legacy access management solutions used in enterprises that would narrowly silo their work. But the risk that the Twitch breach was an insider threat is impossible to ignore.
Developers need modern tools that make it easy to grant privilege, and just as easy to spot who no longer needs it.
Don’t blame the developers
Whatever the root cause(s) of the breach, we need to remember it’s not the collective developers’ fault. I hope the Twitch team approaches this with Norm Kerth’s Prime Directive in mind:
Regardless of what we discover, we understand and truly believe that everyone did the best job they could, given what they knew at the time, their skills and abilities, the resources available, and the situation at hand.
The developers who entered the secrets into the source code were solving the problems they had in front of them, working to delight users. And the reason nobody worked to implement a more secure secret management solution is that nobody responsible for the business goals prioritized that work.
HugOps are needed
Not so much a lesson as a hope for the Twitch team. In addition to identifying the source of the breach, they’ve got a lot of work ahead of them hardening their systems now that this bright spotlight has been turned toward them.
That’s hard work in the best of times, but it’s especially hard amidst the stress of an incident that’s probably disrupted all their normal workflows.