How we built delivery risk detection: the near-miss patterns that actually predict a bad release

Most "risk" features in delivery tools are a red dot next to a due date. That is not risk detection, that is a calendar with anxiety. We wanted something that actually reflected the failure patterns that cause bad releases, which meant starting from the incidents themselves, not from what looked good on a dashboard.

This post is a walkthrough of the actual detection logic behind SignalsAI's risk feed, what it watches for, why, and where we were deliberately conservative about automating a response versus just surfacing a flag.

The pattern that started this: merged code, incomplete QA, one day to release

The single most common near-miss we heard about from engineering leads, independent of company size, was some version of: a PR gets merged into a release branch, QA has not signed off, and nobody notices until the release window is uncomfortably close.

This is a state-tracking problem, not a prediction problem. It does not require a model. It requires the system to actually know three things simultaneously, which most tools track in three different places: the PR's merge status, the task's QA status, and the release date. Cross-referencing those three signals catches the pattern reliably, no ML required. The hard part is not detecting it, it is that most teams' tools do not put those three facts next to each other at all.

We deliberately did not automate the fix here. The tempting version of this feature reverts the PR from the release branch automatically once the flag fires. We did not build that. Reverting from a release branch has real blast radius, hotfix conflicts, broken dependent commits, and getting it wrong is worse than a late flag. The system surfaces the risk loudly and early; a human decides what to do about it. Detection and remediation do not have to live at the same trust level.

Blocked state duration, not blocked state as a boolean

A binary "blocked: yes" flag is nearly useless, because blocked-for-twenty-minutes and blocked-for-four-days are completely different problems, and a boolean cannot tell them apart. Our blocked state carries a timestamp and a reason, and the risk feed surfaces duration, not just presence.

The threshold that matters here is not universal, it is relative to the task's own typical cycle time. A task blocked for six hours might be completely normal for a complex integration and a genuine outlier for a routine fix. We compare blocked duration against the historical distribution for similar work rather than a single hardcoded number, which cuts down meaningfully on alert fatigue from teams whose "normal" blocked time is just different from another team's.

Agent loop detection: the newest failure mode, and the hardest to define

This one did not exist as a category two years ago. An agent "looping" looks, from a distance, exactly like an agent working: tokens are being spent, commits or diffs are being generated, activity is happening. The difference between productive iteration and an unproductive loop is subtle and shows up in the shape of the activity, not its presence.

The signals we found most reliable:

• Repeated attempts at the same file/function with high textual similarity between iterations (the agent is regenerating something close to what it already tried and already failed)
• Cost or token spend materially exceeding the historical median for similar task types, without a corresponding increase in diff size or test pass rate
• Duration exceeding what similar tasks normally take, combined with no state transition (still "in progress" long after comparable tasks would have moved to review)

None of these alone is conclusive, a genuinely hard problem can legitimately take longer and cost more. It is the combination, longer and more expensive and not converging, that reliably separates a stuck agent from a slow-but-working one. We treat this as a flag for a human to glance at, not an automatic kill switch, for the same reason as the release-branch case above: false positives here are cheap (someone looks, sees it is fine, dismisses it), false negatives from an overly conservative auto-kill are expensive (a legitimately hard task gets terminated mid-solution).

Why we chose flags over automation, consistently

There is a pattern across all three of these: detect aggressively, act conservatively. Every one of these risks could theoretically be resolved automatically, revert the PR, unblock the ticket, kill the agent, and every one of those automated actions has a failure mode worse than a human seeing a flag a few minutes later than instant. We think this is the right default for anything touching release branches, in-flight work, or an agent's live session. The cost asymmetry between "flag was noisy" and "automation did something destructive" is not close.

Where we are more willing to automate is read-only synthesis: gathering context, surfacing a suggestion, drafting a summary. Those are cheap to be wrong about. Reverting code, killing a process, or reassigning someone's ticket without their input are not, and we have kept the human in the loop for all three, deliberately, even though it would have been technically straightforward to automate them sooner.

SignalsAI's risk feed runs this detection continuously across every feature in flight.