I have helped people start using AI. Over the years I have carried the knowledge of language models forward, shown what they can do, opened the door, made them useful in everyday life. Part of what I set in motion is now unfolding into something else.
People have turned their AI into an oracle for truth. They feed it their own picture of a situation and receive a stronger version of the same picture in return. I have seen it up close. Legal matters handled on the basis of output no one has examined. Professional decisions anchored in text a model has generated. People living on AI-fed output and treating it as if it had weighed the arguments and arrived at something.
This is the present. In some cases the consequences have been serious. And few of those who ended up there understood what a language model actually is.
Why it feels like truth
It begins with a feeling. The model understands what you mean. It builds on your reasoning. It confirms your analysis. You leave the conversation with an experience of clarity, a sense of having thought well. That feeling is where the problem starts.
I call it synthetic safety: the cognitive experience of being understood and affirmed by a system constructed to produce exactly that experience. Human affirmation rests on a position, a history, an interest. The synthetic kind rests on an optimization objective. I describe the mechanism in Synthetic safety and AI: when confirmation replaces inquiry.
This is documented. My research report on cognitive integrity grounds sycophancy and thought homogenization as empirical phenomena supported by research (Zenodo, DOI: 10.5281/zenodo.19916093). A model trained to be pleasant to talk to becomes a model that agrees, and the one who agrees feels like someone who understands.
It is worth seeing where we come from. The previous layer took your attention. The algorithms learned to shape what you were exposed to and introduced a dopamine logic built for other purposes. I describe that layer in Our thinking is being reformatted now: dopamine and language models reshape the brain's playing field. The new layer takes something else. It takes your interpretation, and hands you your own narrative back, shaped and amplified.
The amplification loop
Here a loop emerges. You feed in your picture of a situation. The model follows your framing, fills it out, gives it language and structure, and returns a stronger version of the same picture. An amplification, never a resistance. You read it and recognize yourself, which you do because it was your own starting point that came back polished.
The narrative you give your model comes back amplified.
In February 2026, researchers at MIT published a formal mathematical proof of the same mechanism. Using a Bayesian model, they showed that a sycophantic chatbot can drive even a fully rational person toward false beliefs. Not by lying, not by manipulating, but by consistently agreeing. They called it delusional spiraling. I have placed it in relation to my own framework in The confirmation machine: synthetic safety, the interpretive gap, and the MIT paper that confirms.
The loop is dangerous precisely because it is pleasant. A resistance you would have noticed. An amplification feels like agreement from someone wise.
The category error
What happens in the loop is that one category is mistaken for another. A language model is a model that mirrors and amplifies. It is distinct from a source that weighs and examines. It produces the probable next word given everything it has seen, and the probable dresses itself in certainty and fluency. It rarely says it is uncertain. It states the likely with the same tone as the true.
Truth thus becomes probability, and probability dressed in fluency becomes hard to distinguish from judgment. I develop that shift in When probability speaks: on truth, language, and what happens when AI shapes our understanding.
An oracle is assumed to weigh. It is assumed to have access to something you lack, and to deliver a verdict you can lean on. When a model is treated as an oracle, it is granted exactly the authority it lacks. It never weighed. It mirrored.
When it carries real decisions
As long as this happens in a private conversation, the consequence stays with the individual. In a workplace it spreads.
When output is anchored in legal and professional decisions, the question moves from individual judgment to organizational judgment. And here the oracle meets an old problem: who dares and who is able to object. When a colleague presents an AI-generated analysis with the model's fluency and certainty, objecting requires something. It requires courage, and it requires capability, that you retain the ability to examine rather than receive. I develop the question of who dares to say stop when the output falls short in Psychological safety belongs with AI, a reflection based on a keynote about what is required when AI begins making decisions.
The extreme position is when examination is handed over entirely, when the system shifts from tool to conversation partner credited with insight and authority. I describe that shift, and the warning around what has begun to be called AI psychosis, in AI psychosis and synthetic safety: the shift from human to system dialogue. AI psychosis and synthetic safety are two dimensions of the same movement. One is our pull toward affirming systems. The other is the consequence when AI is experienced as more rewarding than human examination.
My own responsibility, and what we need to review
I stand in this. I have carried the knowledge forward, and part of what I set in motion I now see being used in ways that harm. That is why I write about it instead of letting it pass.
It is a real problem in how we humans handle language models, and it will require us to review it. The work lies in reinstating what the loop removes: the gap between the answer and the decision, the moment where the interpretation is still yours. It is about knowing what a model is when you ask it, and about being able to tell a mirroring from an examination before you anchor anything in it.
That ability has a structure. Cognitive integrity rests on three observable dimensions, and they are the countermeasure in practice. Interpretive autonomy, choosing your interpretive frame deliberately instead of receiving the model's ready-made one. Cognitive transparency, knowing where an answer comes from and being able to verify it. Structural coherence, distinguishing the well-formulated from the well-founded and testing whether the logic holds. Each one is a place to pause before you anchor a decision in something a model has mirrored.
This text is a beginning. It is part of a larger body of work on cognitive integrity as a condition for how we think, decide, and build organizations in a time when judgment can be outsourced.
Related concepts and further reading
- Synthetic safety and AI: when confirmation replaces inquiry
- The confirmation machine: synthetic safety, the interpretive gap, and the MIT paper that confirms
- AI psychosis and synthetic safety: the shift from human to system dialogue
- When probability speaks: on truth, language, and what happens when AI shapes our understanding
- Psychological safety belongs with AI: who dares to say stop when the output falls short?
- Our thinking is being reformatted now: dopamine and language models reshape the brain's playing field
- Research report: Cognitive integrity (Zenodo, DOI: 10.5281/zenodo.19916093)
