This is a case study from our App Modernisation practice — anonymized and generalized, because the pattern it describes is so common that most companies running a system older than five years will recognize themselves in it.
The short version: a routine infrastructure upgrade silently broke a business-critical workflow in a system that "hadn't been touched." Nobody noticed for days. The fix took an afternoon — but only after a methodical investigation that most teams skip. The real lesson is about why legacy systems fail this way, and what that means if you're planning to modernize one.
The problem: an upgrade that "couldn't" have caused it
The client — a mid-sized distribution business — ran its operations on an internal web application built roughly a decade earlier. Orders, stock levels, and invoicing all lived in this one monolithic system. It worked. Nobody loved it, but it worked.
As part of routine maintenance, their IT provider applied a standard infrastructure update: the database server was upgraded to a supported version, and the operating system beneath it received pending patches. Sensible, overdue housekeeping. The application came back up, users logged in, orders flowed. Everyone moved on.
Four days later, accounting noticed the nightly invoice batch was incomplete. Some invoices generated normally. Others were missing line items. A handful referenced stock quantities that didn't match the warehouse. Nothing had crashed — no error pages, no alerts, no failed jobs in any log anyone was watching. The system had broken silently, in the one workflow where silent errors turn directly into wrong numbers on customer documents.
The immediate response was predictable and understandable: "The upgrade must be rolled back." But a full rollback would have reopened the security exposure the patches fixed — and nobody could say with confidence that rolling back would even cure it, because nobody could explain why the upgrade had caused it.
The investigation: isolate what changed, then what broke
We were brought in at this point, and the first thing we did was slow everything down. Fragile systems punish improvisation; the worst outages we see are usually the second incident, caused by a rushed fix to the first.
Our audit approach in situations like this follows a fixed sequence:
1. Build a change inventory
Before theorizing, list every single thing that changed, however trivial: database version, OS packages, updated system libraries, altered default configuration values, restarted services, even the reboot itself. Upgrades are never one change — they're a bundle of dozens, and the vendor's release notes only describe the headline items.
2. Map the dependency chain of the broken workflow
We traced the invoice batch end to end: what triggers it, what it reads, what it computes, what it writes, and what it hands off. This is where legacy reality diverges from documentation — the batch turned out to depend on a scheduled job outside the application, a shared folder with specific permissions, and an intermediate table that a long-forgotten script repopulated nightly. None of this was written down anywhere. It lived in the system the way habits live in a person.
3. Diff configuration, not just code
The application code hadn't changed in months — which is exactly why "nothing changed" felt true to everyone involved. But configuration had: the upgraded database shipped with different default settings, and one environment file had been regenerated during the update with fresh defaults, quietly discarding a decade-old customization.
4. Reproduce in isolation
We restored a copy of the system to a staging environment, replayed the upgrade step by step, and ran the invoice batch after each step. This converts guesswork into observation: the failure appeared at a specific, repeatable point — and only then did we know we'd found a cause rather than a coincidence.
The root cause: hidden coupling between infrastructure and business logic
In general terms — and this is the part that applies to almost every legacy system — the batch process had been implicitly depending on infrastructure behavior that was never guaranteed:
- It relied on the default ordering and comparison behavior of the old database version when assembling invoice lines. The new version's updated defaults were subtly different — legal, documented, and fatal to logic that had only ever been tested against the old behavior.
- It assumed a legacy compatibility setting that the old server enabled by default and the new one disabled by default. Nobody had ever set it explicitly, so the upgrade silently flipped it.
- Part of the workflow ran as an undocumented scheduled task under an account whose environment changed with the OS patch — so one stage of the nightly chain ran with different permissions and failed quietly, feeding partial data downstream instead of stopping.
Notice what's not in that list: bugs. Nothing here was broken code. Every component behaved correctly according to its own contract. The failure lived in the undocumented seams between layers — the accumulated, invisible agreements between an application and the exact infrastructure it grew up on. That's what a decade of "it works, don't touch it" actually builds: correctness by fossilization.
This is why "we only upgraded the server, we didn't touch the application" is simultaneously true and irrelevant. In a legacy system, the infrastructure is part of the application.
The resolution: controlled overrides, staged rollout, rollback plan
The remediation was deliberately boring — which is the point:
- Fix in staging first. Every change was validated against the replica, with the invoice batch run against known historical data and compared line-by-line to known-good output.
- Explicit configuration over silent defaults. Instead of editing the database's core configuration files in place — which the next upgrade would overwrite again — the required settings were declared in a separate, documented override file, version-controlled, with a comment explaining why each setting exists. The system's implicit assumptions became explicit contracts.
- Make the invisible visible. The undocumented scheduled task was rewritten as a first-class, monitored job with explicit failure alerts. Silent partial failure — the most dangerous behavior in the whole incident — was engineered out.
- Roll out with a way back. The production fix shipped with a tested rollback path and a verification checklist: run the batch, reconcile against staging output, confirm with accounting before calling it done.
Total production downtime for the fix: none. The afternoon of work was cheap. The four days of wrong invoices were not.
The lesson: audit first, modernize second
The uncomfortable truth is that this incident wasn't caused by the upgrade. It was caused by ten years of undocumented coupling, and the upgrade merely collected the debt. Any change — a migration, a new integration, a modernization project — would eventually have triggered it.
That's exactly why our modernisation projects begin with the same methodology we used in this rescue, packaged as a Digital Consultancy audit before any replacement work starts:
- Dependency mapping of the workflows the business actually depends on — including the cron jobs, shared folders, and side scripts nobody remembers.
- Configuration archaeology: which settings are load-bearing, which are accidents, and which will bite during migration.
- A risk-ranked modernization roadmap, so the replacement system is built with full knowledge of the seams — and the cutover runs old and new in parallel until the numbers match.
Modernization done in this order is undramatic. Data moves, nothing is lost, and the business doesn't stop. Done in the reverse order — build first, discover later — the hidden seams surface one by one in production, as urgent incidents instead of line items in an audit report. The same knowledge gets paid for either way; the only choice is whether you buy it calmly or during an outage. If you're planning a rebuild, our App Modernisation process is built entirely around that principle.
Written by Steven Wijaya — Founder at iter8.