Observer Ladder for LLM Introspection Claims Shows Full-Distribution Readers Match Self-Reporters
A new evaluation framework ranks outside observers into sampled text (T), full next-token distribution (D), and weights (W), demonstrating on Qwen3 models that
On November 14, 2026, researchers published an experimental framework designed to test whether large language models (LLMs) possess genuine "privileged access" when reporting on their own internal states. Rather than treating introspection as an all-or-nothing property, the framework formalizes outside observers into a three-rung ladder—sampled text (T), full next-token distribution (D), and model weights (W)—to specify exactly which baseline a self-reporting model must outperform.

Image source: https://rstation.ai/problems/ai_consciousness/papers/5
In prior studies on AI self-awareness and introspection, claims of internal state detection often rested on showing that a model could report an injected or hidden condition that an external reader could not infer from its generated text. The new work demonstrates through a linear-algebra lemma and controlled experiments on Qwen3 models that a massive information gap separates an observer reading sampled words from one inspecting the full probability distribution over the vocabulary.
The Three-Rung Observer Ladder: Text (T), Distribution (D), and Weights (W)
To determine whether an LLM's self-report reflects privileged internal access—information accessible to the model itself but inaccessible to an external predictor—the authors define three progressively stronger classes of outside observers:
- Class (T) — Sampled text: An observer that sees only the discrete tokens actually sampled and emitted as output text. Because sampling collapses the model's entire vocabulary scoring into single chosen tokens, subtle shifts in token probabilities are discarded.
- Class (D) — Full next-token distribution: An observer with access to the complete probability distribution over all vocabulary tokens at every visible position, including positions immediately preceding the assistant's reply. This represents the strongest possible black-box outsider that does not inspect model weights.
- Class (W) — Weights: A white-box observer with full access to model parameters and all intermediate layer activations. Because a Class-(W) observer can inspect everything the model itself computes, outperforming (W) is impossible by definition.
The framework relies on the geometry of the final transformer layer. In the checked Qwen3 models, the centered unembedding matrix—the linear projection mapping normalized final-layer hidden states to vocabulary logits—has full column rank. Consequently, an observer holding the full next-token log-probability distribution at a visible token position can linearly recover the model's final normalized hidden state at that position. For a self-report to beat every linear Class-(D) observer, the internal signal must be attenuated or absent in every visible final-layer state while remaining accessible to the model's own self-reporting mechanism.
Qwen3-0.6B and 1.7B Experiments: Text Near Chance, Distribution at 1.000 AUROC
The researchers tested this boundary by injecting a one-bit signal along a random low-variance direction into layer 14 of Qwen3-0.6B across prompt positions. The injection was calibrated to be "KL-quiet," meaning it produced minimal Kullback–Leibler divergence in output token probabilities so that surface text behavior remained virtually unchanged.
Evaluation across the observer ladder revealed a sharp separation between text-level and distribution-level detection:
- Trained self-reporter (LoRA): Across two random seeds, a LoRA-adapted self-reporting model detected the injected one-bit state at a held-out AUROC of 1.000 (where 1.000 denotes perfect separation and 0.500 denotes random guessing).
- Class-(T) text observers: External text readers evaluating free-running replies generated by the adapted model achieved only 0.531 AUROC, and readers evaluating base-model replies scored 0.523—both near random chance. Re-reading the exact generated tokens with the same weights while the injected state was turned off yielded 0.509 AUROC.
- Class-(D) full-distribution observers: Applying a linear matched filter—a closed-form statistical classifier using Ledoit–Wolf-shrunk covariance—to the final normed hidden states at the three chat-template token positions (
p0–p2) that initiate the assistant turn achieved a held-out AUROC of 1.000 across injection doses from 0.005 to 0.02.
Scaling the same evaluation to Qwen3-1.7B at an injection dose of 0.01 produced the same result: a linear observer reading the initial assistant template position (p0) reached 1.000 AUROC without any access to model weights.
Editorial Analysis and Verified Methodological Boundaries
From an editorial standpoint, placing the near-chance Class-(T) score (0.531 AUROC) side by side with the ceiling Class-(D) score (1.000 AUROC) exposes a key methodological pitfall in evaluating AI self-reports. When an experiment only benchmarks against sampled text, an apparent advantage may simply reflect residual linear traces left on chat-template boundary tokens rather than genuinely private internal monitoring. Requiring introspection claims to specify whether they clear the Class-(D) rung sets a far clearer empirical bar.
At the same time, the authors document specific technical caveats and open questions that bound their findings:
- Linear erasure requires nonlinear Class-(D) readers: When researchers linearly erased the signal direction at the chat-template positions, linear matched filters were blinded (dropping to approximately 0.51 AUROC), while the self-reporter remained unaffected. However, a small multilayer perceptron (MLP) trained on those linearly erased states still achieved 1.000 AUROC, leading the authors to define the Class-(D) ceiling as the maximum of linear and small-MLP readers.
- Nonlinear concealment remains untested: Whether a model could hide an internal state through nonlinear transformations that evade both linear filters and small MLP readers at visible positions was not evaluated in this study.
- Scope of synthetic state injection: All measurements reflect controlled one-bit direction injections in Qwen3-0.6B and Qwen3-1.7B rather than naturally occurring multi-step reasoning states in larger frontier systems.