Grad Detect: Gradient-Based Hallucination Detection in LLMs

The rapid development of Transformer-based large language models (Brown et al., 2020; Vaswani et al., 2017) has led to unprecedented performance in NLP tasks, including question answering, summarization, and reasoning. These models, trained on massive datasets, have demonstrated remarkable capabilities but also exhibit significant limitations, notably the tendency to hallucinate—producing plausible-sounding but factually incorrect outputs. Traditional detection methods rely heavily on output confidence scores (Kadavath et al., 2022), entropy measures (Kuhn et al., 2023), or multiple sampling strategies (Manakul et al., 2023), which are often miscalibrated or computationally intensive. Recent research has shifted focus toward internal model signals, such as hidden states (Azaria & Mitchell, 2023; Ji et al., 2024), to better understand and predict hallucinations. However, these internal representations are snapshots of activations, which may not fully capture the model’s internal sensitivities. Gradient analysis, a technique rooted in optimization and interpretability research (Simonyan et al., 2013; Goodfellow et al., 2015), offers a promising avenue by encoding the entire parameter space’s responsiveness to specific inputs. This paper builds upon these insights, proposing a systematic framework that leverages layer-wise gradients for hallucination detection, aiming for efficiency, interpretability, and robustness.

Despite progress in internal state analysis, existing hallucination detection methods face several challenges. Output-based approaches suffer from calibration issues, as models often assign high confidence to incorrect answers, especially in ambiguous or adversarial scenarios. Multi-sampling methods improve reliability but incur high inference costs, making them unsuitable for real-time deployment. Hidden-state probing provides some internal signals but captures only a static snapshot, lacking the sensitivity to detect subtle internal conflicts that lead to hallucinations. Moreover, current methods often treat correctness and abstention as separate tasks, missing opportunities for unified detection. The core problem, therefore, is to develop an efficient, interpretable, and unified internal signal that can reliably predict hallucinations and model uncertainty during inference without significant computational overhead.

The key innovations of this work include: 1) Introducing a layer-wise gradient analysis framework that captures the internal sensitivities of the entire model, rather than relying solely on output probabilities or hidden states. 2) Constructing category-specific reference gradients through offline averaging over labeled datasets, enabling the formation of prototype vectors in gradient space. 3) Using cosine similarity between sample gradients and reference prototypes to generate discriminative features that encode correctness and abstention signals simultaneously. 4) Employing a lightweight transformer encoder to process the similarity features, learning complex cross-layer dependencies without fine-tuning the large language model itself. 5) Demonstrating that the final five layers contain over 97% of the discriminative gradient information, facilitating efficient detection with minimal performance loss. These innovations collectively enable a scalable, interpretable, and high-performing hallucination detection system that operates in a single inference pass.

• �� Gradient Extraction: For each input query, compute the loss against fixed probe responses (affirming or rejecting) and derive the gradient with respect to model parameters. Focus on the gradients of the MLP down-projection weights within each transformer layer. • Layer-wise Decomposition: Decompose the total gradient into per-layer components, capturing the sensitivity at each level of the model hierarchy. • Reference Gradient Construction: Using labeled data, average the per-layer gradients for each category (correct, incorrect, abstain) and probe response (affirming or rejecting) to form prototype reference gradients. • Similarity Computation: For each sample, calculate the cosine similarity between its per-layer gradient and each reference prototype, generating a matrix of size L×|C|, where L is the number of layers and |C| the number of categories. • Feature Encoding: Flatten the similarity matrix into a sequence, process it with a small transformer encoder that includes positional embeddings, multi-head self-attention, and pooling. • Classification: Use a two-layer MLP to classify the resulting features into the predefined categories, predicting correctness and abstention simultaneously. • Inference: During testing, only a single gradient computation per sample is needed, making the process efficient and suitable for real-time applications.

• �� Datasets: The evaluation spans four benchmark datasets—TriviaQA, SciQ, PopQA, and TruthfulQA—covering factual recall, scientific knowledge, long-tail entities, and adversarial truthfulness. • Models: Eleven instruction-tuned transformer models from four families (Qwen, Falcon, Gemma, and SmolLM), with parameter sizes ranging from 1B to 12B. • Metrics: Detection accuracy (94-99%) and AUC (above 0.85) are primary metrics, complemented by layer ablation studies to assess the contribution of different layers. • Baselines: Compared against confidence scores, sequence perplexity, self-assessment, multi-generation consistency, semantic entropy, and hidden-state probing. • Procedure: Construct labeled datasets with ground-truth annotations, generate probe responses, compute gradients, build reference prototypes, and train the lightweight transformer classifier. • Ablation: Tested performance when using only the last five layers versus all layers, confirming the concentration of discriminative signals in the final layers.

• �� Across all datasets and models, Grad Detect consistently outperforms baselines, with accuracy improvements of 3-8 percentage points over confidence-based methods and AUC gains of 0.05-0.29. • The last five layers contain over 97% of the discriminative gradient signal, enabling high detection performance with reduced computational overhead. • The three-way classification (correct, incorrect, abstain) achieves 72-77% accuracy, demonstrating the method’s ability to encode multiple behavioral signals simultaneously. • The approach maintains robustness across model sizes, with even small models (1B parameters) achieving AUCs above 0.80, significantly higher than traditional output confidence metrics. • Ablation studies show that using only the last five layers results in less than 2% performance loss, validating the efficiency of focusing on these layers.