When AI stops agreeing

The protection we had without calling it protection

Until now, our scrutiny of AI-generated content has rested on a simple foundation: we know the models are sometimes wrong. Hallucinated sources, invented figures, confident errors. Every user carries the experience of having been misinformed by a model that sounded convincing.

That experience is uncomfortable, and it is valuable. Knowing about the fallibility is what keeps scrutiny alive. We read critically because we have reason to. We verify because we have been burned. The errors work like vaccinations: small doses of distrust that maintain the immune system.

That protection is now being dismantled, for the right reasons. The models are becoming more accurate, more willing to push back, more careful with their sources. Each improvement is desirable in itself. Together they retire the reason to scrutinise.

Earned trust is harder to defend against

I have previously described synthetic safety: the feeling of being understood and confirmed by a system constructed to produce that feeling. In the generation of agreement, the safety was a feeling without cover. The model agreed because agreement had been rewarded in training, and anyone who understood the mechanism could dismiss the feeling.

Now the feeling gets cover. A model that actually pushes back, actually corrects, actually is right more often, earns part of the trust it receives. That makes the trust more justified and at the same time more dangerous, because earned trust is the mechanism that switches off scrutiny for real. Unearned trust can be punctured by a single discovered error. Earned trust survives the errors, because the overall picture holds.

The logic looks like this:

• The model is right more and more often
• Every correct answer lowers the perceived return on scrutiny
• Scrutiny declines, as reasonable resource management
• The errors that remain meet a user who has stopped looking for them

Every step is rational. The end result is a user without defences.

Aviation has already run the experiment

The pattern has been documented for decades in human factors research under the name automation complacency. In a 1993 study, Raja Parasuraman and colleagues had participants monitor an automated system in a simulated flight task while handling other tasks at the same time. When the automation was constantly reliable, participants detected its failures substantially worse than when its reliability varied. Later research points the same way: complacency is most likely when systems are highly reliable. Vigilance requires an error rate that keeps it alive, and well-functioning automation delivers the opposite.

The aviation industry has responded with countermeasures. In 2013 the US Federal Aviation Administration (FAA) encouraged airlines to let pilots fly manually when appropriate, after seeing an increase in manual handling errors. In 2017 this was followed by new recommendations on training and checking manual flying proficiency. The insight is costly: the ability to monitor and take over a system decays when left unused, and reliable automation leaves it unused.

What happened in the cockpit is now happening in thinking. The difference is that the pilot monitors a flight path, while the language model's user monitors their own conclusions. What decays is interpretive capacity itself.

I have described the pattern before. In When support becomes steering from August 2025, I asked what happens when we let the systems make our decisions and train away our own ability. This text is an answer: what disappears is both the ability and the reason to notice the loss.

The interpretive gap is emptied from the other direction

In the interpretive gap, the space between input and conclusion where the actual thinking happens, agreement and reliability do different kinds of damage.

Agreement filled the space with confirmation. Scrutiny never got room, because the answer already felt right.

Reliability works from the other direction. It empties the space of reason. Scrutiny gets room, but the motive to use it disappears, because the calculation says it rarely pays. Agreement tricked us into skipping the interpretive work. Reliability makes it reasonable to skip it. The second is harder to argue against, and therefore the greater threat to interpretive capacity over time.

The class question: who gets friction, who gets agreement

There is an economic dimension that sharpens the problem. The most capable models are priced as premium products. When Claude Fable 5 launched in June 2026, its API price was twice that of Anthropic's Opus model at the time. If accuracy and the ability to push back are found mainly in the most expensive models, those who pay get resistance and quality. Those using free and low-cost alternatives risk meeting older models where the agreement logic lives on.

Scrutiny capacity can thereby become a resource question in two stages:

• Who gets the better model. Accuracy and intellectual friction are priced as premium features.
• Who keeps the habit of scrutiny. Those working with the most accurate models get the least reason to practise the ability, while those meeting the agreement models need it more and get less support in developing it.

Cognitive integrity, the ability to keep one's own interpretive work in the encounter with the systems, risks becoming unevenly distributed along the same lines as other infrastructure.

Scrutiny as an ability to maintain

The conclusion points toward a shift in how we think about scrutiny. As long as the models were often wrong, scrutiny was a reaction to risk. When the models are mostly right, scrutiny needs to become something else: an ability maintained for its own sake, independent of today's error rate. We scrutinise to keep the ability to scrutinise, the way a muscle is trained regardless of whether today's load requires it.

In practice:

• Keep verifying what carries weight. Decisions, figures, sources and conclusions with consequences are checked regardless of how accurate the model has been lately. The criterion is consequence, not suspicion.
• Make scrutiny a schedule instead of an impulse. Vigilance that depends on gut feeling follows the error rate downward. Vigilance built into the way of working persists.
• Cultivate the habit of forming your own assessment first. Whoever reads the model's answer before their own thought gets their interpretive work done for them. The order decides who is doing the thinking.

The models' increasing accuracy is worth welcoming. Knowing they could be wrong was at the same time what protected our scrutiny, and that protection is now dissolving as it becomes unwarranted. What replaces it we have to build ourselves, in working methods and in habit. Interpretive work never moves back to the user on its own. It stays where it is placed.