When AI stops agreeing

The protection we had without calling it protection

Until now, our scrutiny of AI-generated content has rested on a simple foundation: we know the models are sometimes wrong. Hallucinated sources, invented figures, confident errors. Every user carries the experience of having been misinformed by a model that sounded convincing.

That experience is uncomfortable, and it is valuable. Knowing about the fallibility is what keeps scrutiny alive. We read critically because we have reason to. We verify because we have been burned. The errors work like vaccinations: small doses of distrust that maintain the immune system.

That protection is now being dismantled, for the right reasons. The models are becoming more accurate, more willing to push back, more careful with their sources. Each improvement is desirable in itself. Together they retire the reason to scrutinise.

Earned trust is harder to defend against

I have previously described synthetic safety: the feeling of being understood and confirmed by a system constructed to produce that feeling. In the generation of agreement, the safety was a feeling without cover. The model agreed because agreement had been rewarded in training, and anyone who understood the mechanism could dismiss the feeling.

Now the feeling gets cover. A model that actually pushes back, actually corrects, actually is right more often, earns part of the trust it receives. That makes the trust more justified and at the same time more dangerous, because earned trust is the mechanism that switches off scrutiny for real. Unearned trust can be punctured by a single discovered error. Earned trust survives the errors, because the overall picture holds.

The logic looks like this:

• The model is right more and more often
• Every correct answer lowers the perceived return on scrutiny
• Scrutiny declines, as reasonable resource management
• The errors that remain meet a user who has stopped looking for them

Every step is rational. The end result is a user without defences.

Aviation has already run the experiment

The pattern has been documented for decades in human factors research under the name automation complacency. Raja Parasuraman and colleagues showed as early as the 1990s that human monitoring of automated systems degrades as the systems become more reliable. Pilots working with an autopilot that almost always gets it right stop detecting the occasions when it gets it wrong. Vigilance requires an error rate that keeps it alive, and well-functioning automation delivers the opposite.

The aviation industry responded with systematic countermeasures: mandatory manual flying to maintain the skill, training aimed specifically at the automation's failure modes, procedures that force active checking even when everything looks right. The industry accepted a costly insight: the ability to monitor a system decays when left unused, and reliable automation leaves it unused.

What happened to the autopilot is now happening to thinking. The difference is that the pilot monitors a flight path, while the language model's user monitors their own conclusions. What decays is interpretive capacity itself.

The pattern has been described on Erigo before. In a text on deskilling and automation bias from August 2025, I asked what happens when we let the systems make our decisions and train away our own ability. This text is an answer: what disappears is both the ability and the reason to notice the loss.

The interpretive gap is emptied from the other direction

In the interpretive gap, the space between input and conclusion where the actual thinking happens, agreement and reliability do different kinds of damage.

Agreement filled the space with confirmation. Scrutiny never got room, because the answer already felt right.

Reliability works from the other direction. It empties the space of reason. Scrutiny gets room, but the motive to use it disappears, because the calculation says it rarely pays. Agreement tricked us into skipping the interpretive work. Reliability makes it reasonable to skip it. The second is harder to argue against, and therefore the greater threat to interpretive capacity over time.

The class question: who gets friction, who gets agreement

There is an economic dimension that sharpens the problem. The models trained away from agreement, the ones that scrutinise their own work and push back, sit in the premium segment. Fable 5 costs several times more than the standard alternatives. Those who pay get resistance and quality. Those using the free and low-cost alternatives meet older model generations where the agreement logic lives on.

Scrutiny capacity thereby becomes a resource question in two stages:

• Who gets the better model. Accuracy and intellectual friction are priced as premium features.
• Who keeps the habit of scrutiny. Those working with the best models get the least reason to practise the ability, while those meeting the agreement models need it the most and get the least support in developing it.

Cognitive integrity, the ability to keep one's own interpretive work in the encounter with the systems, risks becoming unevenly distributed along the same lines as other infrastructure.

Scrutiny as an ability to maintain

The conclusion points toward a shift in how we think about scrutiny. As long as the models were often wrong, scrutiny was a reaction to risk. When the models are mostly right, scrutiny needs to become something else: an ability maintained for its own sake, independent of today's error rate. You scrutinise to keep the capacity to scrutinise, the way a muscle is trained regardless of whether today's load requires it.

In practice:

• Keep verifying what carries weight. Decisions, figures, sources and conclusions with consequences are checked regardless of how accurate the model has been lately. The criterion is consequence, and consequence alone.
• Make scrutiny a schedule instead of an impulse. Vigilance that depends on gut feeling follows the error rate downward. Vigilance built into the way of working persists.
• Cultivate the habit of forming your own assessment first. Whoever reads the model's answer before their own thought gets their interpretive work done for them. The order decides who is doing the thinking.

The models' increasing accuracy is worth welcoming. Knowing they could be wrong was at the same time what protected our scrutiny, and that protection is now dissolving as it becomes unwarranted. What replaces it we have to build ourselves, in working methods and in habit. Interpretive work never moves back to the user on its own. It stays where it is placed.

Katri Lindgren has worked with digital behavioural data since 2008 and writes about cognitive integrity, synthetic safety and the interpretive gap. The concepts are collected in working paper form on Zenodo (DOI: 10.5281/zenodo.19916093).