Engineering

The CVE in your build was public for months before it reached you

Dependency upgrades are the tech debt nobody owns, so they pile up until one becomes an incident. The fix is continuous: watch the advisory, open the PR, run the tests, surface the breaking change.

ASR

Apollo Space Research

Apollo Space

· 11 min read

Open the lockfile of almost any company’s main service and you will find a package three major versions behind, pinned in place by a comment that says # don't touch, breaks the build. Nobody decided to run that version. Nobody chose it. It simply outlived everyone’s attention, and now it sits there, load-bearing, unpatched, and exactly the kind of thing a security advisory will name in six months.

The advisory, by then, will not be news. It will have been public the whole time.

That is the strange shape of dependency risk. The vulnerability that takes you down is almost never a zero-day nobody saw coming. It is an old one, disclosed long ago, that you could have patched on any quiet Tuesday, and didn’t, because patching it was nobody’s job.

The upgrade nobody owns is the one that owns you

Here is the thesis, and the rest of this post is just its mechanism: the upgrade nobody owns is the one that owns you. Dependency maintenance is the purest form of tech debt, invisible while it accrues, catastrophic when it comes due, and assigned to no one in between. The work is not hard. The work is unowned. And unowned work doesn’t get smaller; it waits.

What follows is how to make it owned, not by a person who’ll forget, and not by a quarterly scramble, but by a system that watches continuously and does the boring parts before they stop being boring.

The naive version: the quarterly upgrade sprint

Every engineering team knows the ritual. Once a quarter, someone notices the dependency situation has gotten bad, and a sprint gets carved out to “do the upgrades.” For two weeks, real feature work stops. An engineer wades into a hundred outdated packages at once, bumps versions in a giant batch, and spends most of the time not upgrading but untangling, which of these forty changes broke the test suite, and which of the forty is innocent.

It feels responsible. It is the opposite.

The first failure is timing. By the time the quarterly sprint arrives, every dependency is months stale, so every upgrade is a major jump full of breaking changes, all landing at once. You are not doing forty small, reviewable upgrades. You are doing one enormous, unreviewable one, where a failure in any single package poisons the whole batch and you can’t tell which.

The second failure is worse, and it’s about the window. A vulnerability gets disclosed in week three of the quarter. Your next upgrade sprint is nine weeks away. For nine weeks you are knowingly, or, more often, unknowingly, shipping the vulnerable version, not because you decided the risk was acceptable, but because nobody was watching the feed where the disclosure landed. The quarterly cadence didn’t reduce the risk. It just set the clock on how long you’d carry it.

The third failure is human. The upgrade sprint is the chore everyone dreads and nobody volunteers for. It produces no feature, no demo, no story for the standup. So it slips. “We’ll do upgrades next sprint” is one of the most reliably broken promises in software, and each time it breaks, the batch gets bigger and the next attempt gets more frightening. The upgrade nobody owns is the one that owns you, and the quarterly sprint is just a calendar pretending to be an owner.

Two ways to handle dependency upgrades. On the left, a quarterly sprint lets debt and a disclosed advisory pile up for weeks until one giant unreviewable batch lands all at once. On the right, a continuous watcher catches each advisory and stale package as it appears and opens one small upgrade at a time.

The fix is a loop, not a sprint

The naive cadence treats upgrades as a thing you do. The fix treats them as a thing the system watches.

Start with the watching, because it’s the part the sprint skips entirely. Somewhere in the world, every day, advisory feeds publish new disclosures, package registries push new releases, and projects announce end-of-life dates for the versions you’re running. That information already exists. The question is only whether anything in your company is reading it. In the naive model, the answer is “an engineer, when they remember.” That’s not a watcher. That’s a hope.

A continuous system reads those feeds the way a smoke detector reads the air, constantly, in the background, for one purpose: to fire the moment something it’s watching for appears. A new advisory naming a package in your lockfile. A version you depend on reaching end-of-life next month. A maintained release that closes a hole in the one you’re on. None of these should ever surprise you, because all of them were announced before they mattered.

But knowing is only the first half. A watcher that just files a ticket has moved the work, not done it, and “there’s a ticket for it” is how the unowned upgrade stays unowned. The system has to act.

So the loop continues: the watcher detects the stale or vulnerable package, an agent opens the upgrade as a real pull request, the test suite runs against it, and the result comes back as one of two clean outcomes. Either the tests pass, in which case you have a small, reviewable, already-green upgrade waiting for a human to glance at and merge, or they fail, in which case the system has just told you something precise and valuable: this specific upgrade has a breaking change, and here is the test that caught it.

That second outcome is the one people underrate. A failing upgrade PR is not a problem. It’s the diagnosis you’d otherwise have paid for in the quarterly batch, delivered one package at a time, isolated, with the exact failing test attached.

Why “open the PR” is the whole trick

There’s a version of this idea that already exists and partly works, the automated bot that opens dependency-bump pull requests. It’s a real improvement over the quarterly sprint, and it’s worth being honest about where it stops short, because that gap is the interesting part.

The bot opens the PR. Then it waits for a human, and the human is the same depleted, dread-filled engineer from the quarterly sprint, now receiving the same chore in smaller pieces. A folder of forty open upgrade PRs, each red or yellow, none of them anybody’s job, is just the quarterly batch wearing a different costume. The bot automated the opening. It didn’t automate the judgment, and judgment was always the expensive part.

Here’s the difference. When an upgrade PR comes back green, the judgment is trivial: a human confirms the change is what it claims and merges it. That case should require almost nothing from a person, and in a continuous system it doesn’t.

When it comes back red, the judgment is real, and that’s exactly where the work should concentrate. A breaking change means a behavior changed underneath you. Maybe a function was renamed. Maybe a default flipped. Maybe a return type narrowed and your code quietly depended on the old shape. The naive flow hands you a red checkmark and a stack trace and says good luck. A system built for this reads the failing test, reads the dependency’s changelog, and hands you the actual sentence: this version renamed the call your code uses on line forty; here is the rename, here is the one place it bites. The breaking change stops being a mystery you reverse-engineer and becomes a diagnosis you approve.

That is the move. The point was never to merge upgrades automatically, touching production dependencies without a human is exactly the kind of thing that should stay gated. The point is to do all the boring labor up to the human’s decision: watch the feed, open the change, run the tests, isolate the break, explain it in plain words. The human keeps the one job worth keeping, deciding, and loses every job that was just toil.

A continuous dependency loop. A watcher reads advisory and release feeds, an agent opens one upgrade pull request, the test suite runs, and the result splits two ways: a green change waits for a quick human merge, while a red change comes back with the exact breaking change named and the failing test attached.

This is one watcher among many

Step back and the dependency loop stops looking special. It’s a specific instance of a general shape: something happens in the world, a system notices, and it does the toil up to the point where a human decides.

The advisory feed is one source. But the same loop watches an end-of-life calendar, a license that changed terms in a release, a transitive dependency three layers down that nobody chose and nobody knows is there. Each of these is a thing that, today, somebody is supposed to be watching and nobody actually is, because watching is the kind of work that has no deadline until it suddenly has a very expensive one.

Imagine the surface area. Say a typical service pulls in a few hundred packages once you count the dependencies of your dependencies. No human reads a few hundred changelogs. No human tracks a few hundred end-of-life dates. The reason dependency debt is universal is not that engineers are careless, it’s that the watching exceeds what attention can cover, and attention is the only thing the naive model staffs it with. The upgrade nobody owns is the one that owns you, and at a few hundred packages, nobody is the default owner of almost all of them.

A system doesn’t get tired at package two hundred. It reads the changelog you’d skip. It tracks the date you’d forget. It opens the PR you’d put off. And it does this on the boring Tuesday, months before the advisory becomes news, which is the only time the upgrade is ever cheap.

The turn: tech debt is a question of who’s watching, not how hard you work

Here’s the part that isn’t about dependencies at all.

We talk about tech debt as if it were a measure of how messy the code is, or how rushed the team was, or how much shortcut got taken under deadline. Sometimes it is. But the debt that actually hurts, the unpatched package, the lapsed certificate, the deprecated API that stops working the day the provider finally turns it off, isn’t messy code. It’s unwatched code. It was fine when you shipped it. It rotted because the world moved and nothing in your company was assigned to notice.

That’s a strange thing to ask a person to do well. “Continuously watch a few hundred feeds for the one item that will matter someday” is a job humans are structurally bad at, not from lack of skill, but because vigilance without an event is the most unrewarding work there is. You can’t praise someone for the incident that didn’t happen because they patched the thing nobody else saw. So the diligent engineer who keeps the dependencies current is doing invisible, thankless, easily-deferred labor, and the moment they get busy, or leave, the watching stops and the clock starts.

The promise isn’t a smarter upgrade bot. It’s that the watching stops depending on someone remembering to be diligent. The advisory gets read the day it’s published. The stale package gets a PR before it’s a story. The breaking change arrives already named, on a Tuesday, while there’s still all the time in the world to deal with it, so the most careful person on your team gets to stop being the company’s smoke detector and gets to go build something instead.


This is part of what we’re building at Apollo Space, not a faster way to do the upgrade sprint, but a system that watches the feeds your company can’t, opens the small change before it becomes a large one, and surfaces the break in plain words so a human only ever has to do the deciding. If you’ve ever found a three-versions-behind package pinned in place by a don't touch comment, you already know the upgrade was never the hard part. Remembering to do it was.

Apollo runs your company's repetitive ops so your team doesn't.

Join the waitlist for early access, founding-user pricing, and a front-row seat as we ship.

Join the waitlist