Activation-Based Hallucination Detection in Transformer Decoders

Transformer decoders leak information about their own unreliability well before they finish writing a sentence. That signal sits in hidden states, attention patterns, and gradients, and a growing body of research is learning to read it. The stakes are no longer academic: sanctions for AI-hallucinated legal filings jumped from roughly $5,000 in a single 2023 matter to $55,597 in a single 2025 matter, an increase of about 11 times in a year and a half. Meanwhile, on the FACTS benchmark, a multi-dimensional test of factuality, no model clears 70%. The leading score, from Gemini 3 Pro, is 68.8%, meaning even the best system on record is still wrong on close to a third of what it's asked. Fluent, grammatically confident prose gives no clue as to which third that is. Detection work has moved inside the model because of this.

What "white-box" detection means

Most hallucination-detection methods built on model output share a dependency that limits their use in practice. Reference-based approaches need ground truth to check against. Consistency-based approaches need multiple generations from the same prompt to check for self-contradiction. Both cost time, compute, or access to data that simply isn't available when a system is answering a live query.

Hidden-state-based detection removes that requirement. It works from a single forward pass, needs no external reference, and needs no resampling. That's the practical case for "white-box" detection: reading what the model already computed, rather than paying twice for a second opinion.

Language models encode signals of truthfulness in their internal representations, separate from whatever text eventually comes out, and this premise has held up across a fair amount of research. Nobody disputed that the signal exists; extracting it reliably, and as early as possible, was always the harder task. It's how to extract it reliably, and how early. Early work in this space mostly answered the first half of that question and stopped there, flagging errors after the fact without explaining how the signal was represented internally or whether it could be used further upstream, before a token ever gets committed.

Linear probing: the foundational approach and its known ceiling

SAPLMA set the template for the field. It trains a simple feedforward classifier, an MLP, on hidden states pulled from specific layers of the model, and the result was striking on its own terms: internal representations carried a real truthfulness signal. A single mid-layer's mean-pooled activations were enough to hit 84% accuracy, 91% AUC, and an F1 score of 75%. That's a real signal, extractable with a classifier about as simple as it gets.

But the approach has a ceiling baked into its design. Linear probing at one layer and one token position throws away everything happening at other layers and other steps in generation. Worse, the layer where the signal peaks isn't fixed. It shifts depending on the dataset and depending on the model, so there's no single probe position that generalizes across settings.

MHAD looks for neurons and layers that show hallucination awareness at both the start and the end of generation. It concatenates the outputs of those selected neurons into what its authors call a hallucination awareness vector, and the reported result is an improvement over prior detection methods across several different LLMs. The lesson from SAPLMA to MHAD is really about scope: a wider net across layers and generation steps catches more of the signal than any fixed point can.

Beyond single-layer snapshots: dynamic, attention-graph, and full-tensor approaches

Some of the more recent work abandons the static-snapshot idea. The ICR Probe treats hallucination detection as a question about the process of updating hidden states, not the states themselves. It introduces something called the ICR Score, short for Information Contribution to Residual Stream, which measures how much each module in the network contributes to updating the hidden state as information flows through it. The probe then aggregates those scores across layers, which lets it capture how the representation evolves over the full depth of the network rather than freezing it at one point. Despite tracking more of the network's behavior, it does so with fewer parameters than earlier probes.

The underlying diagnostic logic uses Jensen-Shannon divergence, comparing the projection of the hidden-state update against attention scores layer by layer, to isolate how much each module is contributing to the residual stream at that step. Fluctuations in that measure serve as the diagnostic signal for potential hallucination.

TOHA takes a different route entirely, built specifically for retrieval-augmented generation, where the target isn't abstract truthfulness but faithfulness to a retrieved document. It builds a graph out of the attention matrices and computes a topological divergence metric over that graph. Higher divergence in particular attention heads tracks with unfaithful output, and that relationship holds across datasets, which is a meaningfully stronger claim than most single-benchmark results in this space.

Other approaches occupy the space between single-layer probes and full-tensor methods, attempting to capture richer signals while keeping computational overhead manageable.

Gradient-based detection: a complementary fingerprint from the parameter space

Everything above works from activations: a snapshot of what the model has already computed at a given layer. Gradients are a different kind of signal. They measure how sensitive every parameter in the network is to the current prediction. This means they capture not what the model computed, but what it would need to change to compute something else. That's a much higher-dimensional fingerprint, reflecting the relationship between the entire parameter space and a single output.

Grad Detect operationalizes this with a single forward-backward pass at inference time, analyzing gradient patterns layer by layer. Ablation studies across eleven models spanning four architectural families turned up a fairly clean result: over 97% of the discriminative gradient signal concentrates in the final five layers. That's a big deal for anyone trying to deploy this kind of detection, because it means computing gradients across an entire network can mostly be skipped. Watching the last five layers gets you nearly all of the signal at a fraction of the cost.

The conceptual payoff is that gradients expose instability that activations alone can hide. A hidden state can look perfectly ordinary at a given layer while the gradient underneath it signals that a small change in parameters would flip the prediction. Activations show the computation. Gradients show the fragility beneath it.

Streaming detection in chain-of-thought reasoning as a special case

Chain-of-thought reasoning breaks the assumptions the earlier methods were built on. Errors in a long reasoning trace don't usually announce themselves. They emerge subtly, and they propagate: an intermediate step can look locally plausible and internally consistent, even as the whole trajectory drifts toward a wrong conclusion. The individual steps are points along a single evolving trajectory, not independent events to be judged one at a time. They're points along a single evolving trajectory, and treating hallucination in CoT as a one-off error at a fixed position misses that structure.

The better framing treats hallucination as a kind of evolving latent state rather than a discrete event. That leads to a two-level detection architecture: step-level judgments capture local evidence, flagging what looks off at each individual point in the chain, while a cumulative prefix-level signal integrates those judgments across the whole trajectory so far, tracking the reliability of the reasoning as a whole rather than any single step. This approach was developed against a dataset of more than 10,000 long CoT trajectories, comprising over 200,000 individual reasoning steps, of which more than 40,000 were hallucinated. That scale matters. It's what makes a two-level, streaming architecture viable rather than speculative.

What the detection–correction asymmetry reveals about where these signals live

A finding here should temper any enthusiasm about turning detection straight into correction. Linear probes do detect hallucination signals with above-chance accuracy in larger models. But when researchers tried activation steering, nudging the model's internal state along the direction the probe identified, to actually fix the hallucination, it failed in all seven of seven models tested. Whatever the probe is picking up on, it isn't a lever the model can be pushed on directly.

Detection strength itself is uneven, too. Output-confidence baselines, the much simpler approach of just looking at how confident the model's own output distribution is, actually beat activation probes on raw detection AUC for every model above a moderate parameter threshold. At Pythia-6.9B, that gap reached 0.157 AUC in the confidence baseline's favor. The case for probing is that it's the only one available before the model has produced anything: probe signals are readable at position zero, before a single output token exists, which is structurally impossible for any... It's that it's the only one available before the model has produced anything: probe signals are readable at position zero, before a single output token exists, which is structurally impossible for any method that has to look at output.

Even that early signal is inconsistent. It reached statistical significance in only two of the seven models tested, Pythia-1.4B (p = 0.012) and Qwen-2.5-7B (p = 0.038), and it was absent entirely in the smaller models and in the base-only Pythia-6.9B. The pre-generation signal appeared with statistical significance in only two of the seven models tested, Pythia-1.4B and Qwen-2.5-7B, and was absent in the smaller models and in the base-only Pythia-6.9B, so it is not yet strong enough to stand alone as a production-grade detector.

Why benchmark results in this field should be read with caution

A large evaluation known as PARALLAX tested twenty-two detection methods across twelve open-source models spanning six architectural families and six corpora, and the results should give anyone citing a single benchmark number some pause. Much of what looks like progress in this field doesn't hold up once the evaluation setup is scrutinized closely.

Part of the problem is leakage. Several of the commonly used corpora embed information that makes detection artificially easy. A baseline called TxTemb, which does nothing more sophisticated than text similarity and has no access whatsoever to model internals, exploits that leakage to post near-perfect detection scores. No white-box signal required, because the benchmark handed the answer to the detector on its way in.

Generalization fares no better. Even supervised probes that perform adequately within one evaluation regime tend to collapse when pushed across different corpora or model families. The signal such probes learn is tied to the specific setup they were trained on.

That matters for reading every method discussed above. A high AUC score on one benchmark, on its own, proves very little. The tests that actually mean something are cross-corpus, cross-regime, and cross-model transfer, and those are exactly the tests most published results have not yet passed.

Agentic and tool-calling settings as the emerging deployment frontier

Agentic systems, where an LLM doesn't just generate text but selects and invokes external tools, introduce failure modes none of the methods above were built to catch. Models can hallucinate at the level of tool use rather than text generation. They generate malformed parameters, arguments that are structurally invalid for the tool being called. And in some cases they engage in what's been termed "tool bypass": simulating what a tool's output would look like instead of actually invoking the tool, which quietly defeats the audit trail and security controls the tool call was supposed to go through.

The detection approaches surveyed here were developed against text generation tasks rather than agentic action selection. Extending them into agentic settings, where the thing being hallucinated is an action rather than a sentence, is the frontier this research is only beginning to reach.