How Do Companies Like Facebook Keep Their Software Engineers In Check

Facebook

Facebook employs over 10,000 engineers.

Deploying anything to production isn’t as simple as a rogue engineer secretly hacking something in or out leading to billions of people rendering the end result.

  • Lots of people monitor the code base (since they work on it). You have to familiarize yourself with the flow and the ongoing changes to avoid an unexpected regression you would cause due to parallel changes elsewhere.
  • All changes get QA’d heavily, both via tons of automated tests verifying the core features and a bunch of manual QAs.
  • Code reviews are a standard procedure before deployments (in most companies). Certain folks vet the code base regularly.
  • Changes aren’t rolled over everywhere at the same time at scale. It’s somewhat gradual, starting with test instances to “beta” groups to other small chunks until everyone is migrated.

Basically, you need to organize a sophisticated black operation of multiple people in different seniority working together to cover for a flaw, able to disappear immediately after before the law enforcement gets to them.

This is hard to do (considering recruitment background checks etc), not guaranteed (rotations around people who do reviews or tests), and severely expensive. And even if successfully deployed, version control would allow for reverting this within hours, if not minutes.