The hidden cost of losing architectural knowledge in software projects.

Systems are not lost to failure, but to forgetting. When decisions cannot be undone, change becomes collapse, and rebuilding becomes inevitable.

Every system eventually invites a rewrite.

Almost never because it broke. Usually because nobody left on the team can change it without holding their breath.

The structure feels rigid. The decisions feel outdated. The system resists modification in ways nobody can quite explain, and nobody particularly wants to be the one to justify. A new feature means touching six files that shouldn’t be related to it. A fix in one corner breaks something two teams over, for reasons that take a week to trace. At some point somebody says the quiet part out loud in a planning meeting, and a clean slate starts to look reasonable. Sensible, even. Overdue.

Greenfield work makes a good pitch. No history to work around. Every decision made on purpose this time, by people who finally understand the problem the old system was only ever groping toward. I’ve made this pitch myself, in rooms where everyone was nodding before I finished the sentence. What it delivers, most of the time, is amnesia with a deadline attached.

A rewrite never starts from zero, whatever the kickoff deck says. It starts from assumptions about what the old system did and which of those things mattered, made by people standing outside the system looking in. That’s exactly the vantage point from which those assumptions are hardest to check.

What a mature system is made of

A mature system is more than its code. It’s accumulated truth about an environment that wasn’t the most amenable.

A workaround here, a strange boundary check there, a comment that just says “don’t touch this, see the incident from three years ago.” Each one is a lesson the system learned the hard way, usually at somebody’s expense. Some of that knowledge is written down somewhere findable. Most of it isn’t. It exists as shape: the specific, ugly contour of the code, not something you could hand to a new hire and call documentation.

When a system gets rewritten, that shape doesn’t transfer. It gets rediscovered, piece by piece, the hard way: in production, by whoever’s on call the week some corner case that used to be handled quietly shows up unhandled instead. The new system looks simpler on launch day because it’s forgotten what made the old one complicated. Nobody’s told it yet.

In April 2000, Joel Spolsky wrote up what happened after Netscape decided its browser codebase had become too messy to keep extending, and rewrote it from scratch instead. Version 4.0 shipped in 1997. The rewrite, which became Netscape 6, didn’t ship until late 2000: three years with no serious competitive release, in a market where Internet Explorer was being bundled into every copy of Windows for free. Spolsky’s point wasn’t really about elegance. The ugliest parts of a mature codebase are usually ugly because they encode a fix for a real bug, a workaround for a browser inconsistency nobody remembers reporting, sometimes just a piece of knowledge paid for with a support ticket or an outage. Thrown out in a single motion, in the name of starting clean, because greenfield feels like progress in a way that patching an old system quietly never does. Netscape never recovered the ground it lost in that gap: three years didn’t buy the team clarity, it bought Microsoft a market.

The cost of amnesia

By the time the FBI’s Virtual Case File project collapsed, the cost of amnesia had an actual dollar figure attached to it, and a congressional report to go with it.

In 2001, the FBI contracted with Science Applications International Corporation to replace its paper-heavy, mainframe-backed case-management process with a modern application that would let agents and analysts share investigative information electronically across field offices. On paper, this was overdue and obviously correct. The bureau was still moving physical case files by hand in some offices, years after most private-sector organizations had digitized equivalent workflows. But the system being replaced held decades of procedural knowledge about how casework moved, how highly nuanced evidence chains needed documenting to be held up in court, and how different units needed different, sometimes conflicting views into the same case. Some of that compartmentalization between field offices existed for reasons that had nothing to do with software and everything to do with the nature of investigations. None of it lived in a specification. It lived in how people used the old system and worked around its limits, and it had to be rediscovered by a team building the replacement from outside, lacking any contact with that experience. The FBI’s own understanding of what it needed kept shifting mid-build, because nobody had fully externalized what “what it needed” meant.

By April 2005, the bureau scrapped the project outright. SAIC had delivered roughly 700,000 lines of code, and most of it turned out to be unusable: an estimated $105 million of the total $170 million spent produced nothing the FBI could put into service. IEEE Spectrum’s postmortem is worth reading in full for the granular version. It speaks of continuously shifting requirements, no strong technical leadership on the customer’s side of the table, and a build that tried to leap directly to a finished system instead of growing into one through anything resembling incremental delivery.

The FBI eventually did get a working case-management system, from a follow-up effort called Sentinel, delivered in 2012 and built in phases with far tighter scope discipline and closer government oversight of the contractor relationship. The lesson wasn’t that government can’t build software.

Nobody, inside government or out, can rewrite a system whose real specification exists nowhere but inside the system being replaced.

When greenfield discards the product, not just the code

Digg relaunched as a total rebuild on August 25, 2010, losing the reason people used the product along with the technical knowledge behind it. It had a new design, a new codebase, and new infrastructure. The entire greenfield landed in one release, top to bottom, presented as a clean and overdue modernization. In the process, the team dropped or buried features that had nothing to do with performance and everything to do with what made Digg feel like Digg to the people who used it daily: the “bury” button that let the community collectively downvote content, the “upcoming” page where unproven stories could rise on their own before reaching the front page, the personal history and friend-tracking features longtime users had built habits around. None of them registered as a necessity, or even a consideration, during the rewrite. It also changed who got distribution, giving major publishers a mechanism to auto-submit content directly to the site, which quietly undercut the community-driven ranking that had made Digg worth visiting over any other aggregator. Within about a month, Digg had lost close to 30% of its global traffic, with US traffic down by 26% and UK traffic down by 34% according to some contemporary estimates. A large share of that departing audience went straight to a much younger competitor called Reddit, which had spent years copying the parts of Digg’s original mechanic that the rewrite had just thrown away. Digg laid off about 40% of its staff in the aftermath, and Kevin Rose, the founder, resigned not long after. Search Engine Land’s contemporary account of the relaunch reads, in hindsight, like a checklist of nearly everything a community product should never do to itself in a single release.

The Digg team mistook the product’s implementation for the product itself, and underestimated the migration because of it. They rewrote the former with real technical skill, and discovered six brutal weeks later that the latter had been quietly living inside the parts they’d thrown away.

Longhorn, the internal codename for what eventually shipped as Windows Vista, played out the same failure in slow motion, without a public collapse but with a comparable cost in time. It accumulated features and ambitions for years without anyone drawing a hard boundary around scope, until by mid-2004 it was so far behind that Microsoft reset the project outright, discarding most of what had been built and restarting from the more conservative Windows Server 2003 codebase with a narrower, more defensible target. Vista finally shipped in November 2006, more than five years after Windows XP, at the time the longest gap between major Windows releases in the company’s history. The reset was closer to a correction than a rewrite in the Netscape sense. Nobody at Microsoft was disgusted with the old code, exactly; it just arrived years too late, with the company paying twice: once for the original effort, once to partially undo it before building the real thing.

Reversibility is the missing piece

Reversibility is what actually matters here: a property built into a system deliberately, long before anyone needs it, not something a young codebase gets for free.

A reversible system absorbs change instead of avoiding it. A decision can be revisited without the whole structure coming down around it. A constraint can be loosened without triggering a reconstruction project that eats a year of the roadmap. The system keeps its history instead of severing it every few years out of frustration.

That takes restraint, and restraint is unglamorous, because it rarely produces a demo anyone gets excited about. Reversible decisions are frequently less expressive than the alternative. They avoid locking in an abstraction before anyone’s sure it’s the right one, and they deliberately leave gaps where certainty doesn’t exist yet. From the outside, that can look like indecision, or a team that can’t commit to an architecture. More often it’s just a refusal to encode assumptions the team can’t defend today, and almost certainly won’t be able to defend in two years either.

Twitter’s original backend, written in Ruby on Rails, became infamous for outages severe enough that the fail whale error page turned into a cultural mascot for the whole Web 2.0 era. The company’s Ruby-based message queue “hit a wall” as early as 2008, in the words of one Twitter developer at the time, unable to keep up with long-running, memory-intensive work at scale, under exactly the kind of pressure that tends to produce a Netscape-style decision. The obvious, tempting move was the Netscape move: declare Rails the problem, freeze feature work, rewrite the whole backend from scratch. Twitter didn’t do that. Engineers migrated pieces of the backend onto a new stack built on the JVM, largely in Scala, component by component, over several years, eventually reconfiguring the system so that all mobile traffic bypassed the Ruby stack entirely. The proof came on election night in November 2012, historically one of Twitter’s highest-traffic events. Users posted at a sustained average of nearly 10,000 tweets per second for a full hour, peaking at 874,560 tweets in a single minute. There was no clean launch event, no dramatic relaunch post. The service just held, without the fail whale making an appearance even once, the entire time it was being rebuilt underneath its own users.

Stripe’s internal implementation changes constantly. What doesn’t move is the promise made to everyone who’s already integrated against an older version of the API: engineers write against the current internal version only, and a compatibility layer transforms each outgoing response into whatever version a given client originally requested, sometimes years out of date.

Stripe has described this as treating the API itself as infrastructure, something that has to keep working for people who built on it years ago no matter how much has changed underneath since. The public API hasn’t needed a breaking rewrite in more than a decade.

Around 2015, Segment split a core piece of its product into more than 140 separate services, following the fairly common belief at the time that this was simply how serious companies build at scale. The promised velocity never showed up. Instead, three full-time engineers ended up spending most of their working hours just keeping the constellation of services alive, and a single change to a shared library meant coordinating redeployment across all 140-plus of them. In 2018 the team wrote publicly about collapsing that architecture back into a single service. Deploys that used to require coordinating dozens of independently versioned services now took one engineer a few minutes, alone. Nobody had designed reversibility into the original split. Segment simply treated the decision itself as something that could still be undone once it was clearly a mistake, and corrected it once enough people were willing to say so plainly.

Shopify never made the split in the first place. Facing the same pressure every sufficiently large monolith eventually faces (in Shopify’s case, a Ruby on Rails codebase worked on by more than a thousand engineers across more than a decade), the company seriously considered breaking the monolith into microservices and concluded the operational cost wasn’t worth what it would buy them. Instead, it built an internal tool called Packwerk to enforce hard boundaries between modules living inside the same codebase, giving individual teams the isolation benefits normally associated with separate services: clear ownership, enforced interfaces, the ability to reason about one module without loading the entire system into your head at once, without paying for separate deploys, separate on-call rotations, or the failure modes that come standard with a distributed system. Shopify’s own engineering team has described this as ongoing, unglamorous maintenance rather than a finished migration with a launch date; their account of the effort reads less like a triumphant rewrite story and more like a maintenance log that never quite ends, which is closer to how large systems stay healthy in practice. Nobody had to freeze feature work for a year to make any of it happen.

This is close to what Martin Fowler formalized back in 2004 as the strangler fig pattern. It’s named for a vine that germinates in the canopy of a host tree and gradually grows down and around it, eventually taking the host’s place without the tree ever being felled outright in one motion. Route traffic to new code piece by piece. Let the old system keep running underneath while it’s dismantled from the edges inward, module by module, service by service. Every step stays small enough to reverse if it turns out to be the wrong step, and the system never once stops being live while the work happens around it.

When a rewrite is earned

Starting over isn’t always the wrong call, either. Greenfield becomes necessary the moment reversibility is already gone. Once decisions can’t be undone without triggering cascading failure, or constraints have grown so entangled that no single one can be pulled loose without dragging the others with it, a greenfield rewrite is earned. It’s an admission that the system stopped being changeable some time ago, and the rewrite is just the moment everyone in the room finally says so out loud instead of stalling for one more quarter.

The Netscapes and the Diggs and the Virtual Case Files of the world had already lost the ability to change safely, long before anyone rewrote anything. The rewrite just finished the job of discarding what still worked along with what didn’t, because from inside a system that’s stopped being legible, it’s hard to tell which is which until it’s too late.

In the end, a greenfield rewrite shouldn’t be a choice made on impulse. It should follow the swan song of a system that held onto its ability to change for as long as it could.

The systems that deserve to last are the ones that treated rebuilding as a last resort instead of a first instinct, and that’s a property you have to design for, deliberately, long before the day you’d need it.

Sources

• Joel Spolsky, “Things You Should Never Do, Part I,” Joel on Software, April 2000.
• “It’s official: FBI scraps $170M Virtual Case File project,” Computerworld.
• Harry Goldstein, “Who Killed the Virtual Case File?,” IEEE Spectrum.
• “What Happened to Digg? The Rise, Fall, and Surprising Comeback Story,” Startup Booted.
• “Digg v4: How To Successfully Kill A Community,” Search Engine Land, 2010.
• “Looking back at Windows Longhorn,” The Register, 2025.
• “Twitter survives election after Ruby-to-Java move,” The Register, November 2012.
• “APIs as infrastructure: future-proofing Stripe with versioning,” Stripe.
• “Goodbye Microservices,” Twilio/Segment Engineering.
• “Deconstructing the Monolith: Designing Software that Maximizes Developer Productivity,” Shopify Engineering.
• “Under Deconstruction: The State of Shopify’s Monolith,” Shopify Engineering, 2020.
• Martin Fowler, “StranglerFigApplication,” 2004.