Natural Ungrokking: Asymmetric Control of Which Rules Survive Pretraining
This study reveals the asymmetric dynamics of rule survival in pretraining models, with support frequency predicting rule fate and demonstrating irreversibility via editing interventions.
Key Findings
Methodology
This paper employs a transformer-based language model (11.5M parameters) trained on two corpora—TinyStories and a filtered web corpus—using a systematic experimental framework. The core methodology involves measuring support frequency of rules (e.g., pronoun-gender rule), behavioral evaluation via conflict and agree conditions, and mechanistic analysis through contrast margin (CM). The experiments include support-contradiction editing, where the support evidence is flipped at various ratios (p=0.437, 0.667, 1.0), to causally test the impact on rule survival. Multiple model scales (70M to 1.4B parameters) are tested to verify the universality of the phenomena. The approach combines pre-registered metrics, blind evaluation, and cross-model validation, ensuring robustness and reproducibility.
Key Results
- Support frequency is a key predictor: rules with support frequency above 0.8 in TinyStories consistently survive (9/9), whereas in web corpora, rules with support below this threshold collapse (0/9). Support-contradiction editing demonstrates a strictly monotonic dose-response: increasing the ratio of support flips causes a linear decline in rule performance, with the conflict accuracy dropping from 0.94 to near zero. The mechanism analysis shows that rule collapse coincides with the contrast margin crossing zero, indicating displacement rather than erasure. Larger models tend to have shallower collapses, with the depth inversely related to model size (Spearman ρ=0.894). Even support levels 450 times higher than natural support fail to restore collapsed rules, confirming the irreversibility of the process.
- The collapse mechanism is characterized by displacement: the rule is outcompeted by surface patterns (e.g., default pronoun bias), with the internal representation shifting accordingly. The contrast margin (CM) serves as a reliable predictor, crossing zero within 100 steps of behavioral collapse. The internal circuit analysis reveals that the last-layer heads carry the rule preference, and their reformation correlates with partial circuit reassembly during support injections. The asymmetry is clear: destroying a rule via support editing is effective and monotonic, but restoring it by increasing support is ineffective once collapsed.
- Support frequency causally determines rule fate: causal interventions that flip support evidence in training data produce predictable, monotonic effects on rule survival. The experiments confirm that support support ratio is sufficient to cause rule destruction, but support support ratio alone cannot recover a rule once it has collapsed, demonstrating a fundamental irreversibility. This asymmetry is consistent across different model sizes and datasets, indicating a general principle of rule dynamics in neural language models.
- Model scale influences collapse depth: smaller models (e.g., 70M) exhibit deeper collapses, while larger models (1.4B) tend to have shallower, more superficial collapses. This trend is consistent across different corpora and experimental conditions, suggesting that model complexity and capacity modulate the stability of learned rules. The phenomena are transferable across out-of-distribution datasets, confirming the robustness of the support frequency mechanism.
- Out-of-distribution transfer experiments show that the emergence-and-collapse pattern persists beyond training data, with the same support frequency thresholds predicting rule survival. The experiments demonstrate that the internal mechanism—displacement—is a general phenomenon, not limited to specific datasets or models. This broad applicability underscores the importance of support frequency as a fundamental variable in understanding rule stability.
- The internal mechanism analysis reveals that the rule collapse is associated with a zero crossing of the contrast margin, which occurs within 100 steps of behavioral failure. Circuit-level analysis indicates that the last-layer heads encode the rule preference, and their reformation during support injections correlates with partial circuit recovery. These findings provide mechanistic insights into how rules are internally represented and displaced, emphasizing the importance of internal representations over static outputs.
- The asymmetry in rule control is striking: while rule destruction via support editing is straightforward and monotonic, rule restoration through support augmentation is ineffective once collapse occurs. This highlights the intrinsic fragility of learned rules and the difficulty of reversing internal displacement. The findings suggest that rule stability depends critically on the support frequency, which acts as a causal lever for rule destruction but not for recovery.
- Predictive validity of the mechanism: the zero-crossing of the contrast margin reliably precedes behavioral collapse, serving as an early warning signal. This temporal alignment supports the causal interpretation that internal displacement causes rule failure. The approach offers a quantitative framework for diagnosing and controlling rule dynamics in neural models, with potential applications in model safety and interpretability.
- Cross-model consistency: experiments across multiple model sizes and architectures confirm that support frequency governs rule stability universally. The phenomena are robust against variations in training corpus, model capacity, and out-of-distribution data, establishing a general principle of rule dynamics in neural language models.
Significance
This research fundamentally advances our understanding of how neural language models acquire, maintain, and lose rules. By identifying support frequency as a key predictor and mechanistic driver, it shifts the paradigm from static memorization to dynamic competition and displacement. The findings reveal that rules are inherently fragile, subject to rapid displacement by surface patterns, and that their collapse is predictable and controllable through causal interventions. This has profound implications for AI safety, interpretability, and continual learning, as it provides a quantitative framework to diagnose and mitigate rule loss. Moreover, the universality of the mechanism across models and datasets suggests that rule stability is governed by fundamental principles of neural dynamics, opening avenues for designing more robust models. The ability to predict and manipulate rule survival enhances our capacity to develop AI systems that are transparent, controllable, and aligned with human values, addressing long-standing challenges in AI safety and trustworthiness.
Technical Contribution
This paper introduces a novel mechanistic framework for understanding rule dynamics in neural language models, centered on support frequency as a causal variable. The key contributions include: • Formalizing the support frequency as a predictor of rule survival, validated across multiple datasets and model scales. • Demonstrating that rule collapse occurs via displacement, characterized by the zero crossing of the contrast margin (CM), which is linked to circuit-level changes in the last-layer heads. • Developing a causal intervention methodology that manipulates support evidence in training data, establishing a monotonic dose-response relationship for rule destruction, but an irreversible barrier for rule recovery. • Extending the analysis across out-of-distribution datasets, confirming the universality of the displacement mechanism and support frequency’s predictive power. • Providing a comprehensive mechanistic account that integrates behavioral metrics, internal circuit analysis, and causal interventions, offering a new paradigm for rule management in neural models.
Novelty
This study is the first to systematically demonstrate that rule survival in neural language models is governed by a support frequency law, with a clear causal link established through targeted editing interventions. Unlike prior work focusing on static memorization or surface pattern correlations, this research reveals a dynamic displacement mechanism—rules are displaced by surface patterns rather than erased. The use of causal interventions to manipulate support evidence and observe predictable, monotonic effects on rule collapse is a key innovation. Additionally, the cross-model and out-of-distribution validation underscores the generality of the findings, marking a significant advance in understanding the internal dynamics of rule learning and forgetting in neural networks.
Limitations
- The current experiments focus primarily on simple, syntactic rules such as pronoun gender and article allomorphy. The applicability to more complex semantic or multi-hop reasoning rules remains untested, requiring future exploration.
- Support frequency is measured based on predefined criteria within specific corpora; its robustness across diverse datasets and tasks needs further validation, especially in real-world, noisy environments.
- The causal interventions are limited to support editing; other potential mechanisms of rule collapse, such as internal representation shifts unrelated to support frequency, are not yet explored.
- The models studied are relatively small (up to 1.4B parameters); whether the observed phenomena scale to larger, industrial-grade models requires further investigation.
- The irreversibility of rule collapse, once support is removed, raises questions about the potential for partial recovery through alternative mechanisms, which remains an open area for future research.
AI Executive Summary
Understanding how neural language models acquire, maintain, and sometimes unexpectedly lose rules is crucial for advancing AI safety and interpretability. Traditional views suggest that once a rule is learned, it remains stable unless explicitly forgotten. However, recent findings challenge this assumption, revealing a complex dynamic where rules can suddenly collapse during training, even when the training data continues to support them.
This study investigates the internal mechanisms behind rule collapse, focusing on the role of support frequency—the rate at which training data supports a given rule. Using a transformer-based language model trained on two corpora, TinyStories and a web-derived dataset, the researchers systematically measure how often rules like pronoun gender agreement appear in the training stream. They find that rules with higher support frequency are more likely to survive until the end of training, while those with lower support tend to be displaced by surface patterns, such as default pronoun biases.
A key innovation is the causal manipulation of support evidence through targeted editing. By flipping support tokens at various ratios, the researchers establish a monotonic dose-response relationship: increasing support support ratio causes a predictable decline in rule performance, confirming that support frequency is a causal driver of rule collapse. Conversely, attempts to restore collapsed rules by injecting support at levels far exceeding natural support levels fail, demonstrating an inherent irreversibility—once displaced, rules cannot be simply re-supported to recover.
Further analysis reveals that the collapse mechanism is characterized by displacement rather than erasure. The internal representations, especially in the last-layer heads, shift in response to surface pattern competition, with the contrast margin crossing zero just before behavioral failure. This mechanistic insight is validated across multiple model sizes and out-of-distribution datasets, emphasizing the universality of the phenomenon.
The findings have profound implications. They suggest that rule stability in neural models is governed by a fragile support frequency law, which can be manipulated causally but is inherently difficult to reverse once broken. This insight opens new avenues for designing more robust, controllable AI systems, capable of maintaining critical knowledge in dynamic environments. It also provides a quantitative framework for diagnosing and intervening in rule-related failures, advancing the field toward safer and more interpretable AI.
Deep Analysis
Background
The evolution of neural language models has transitioned from simple pattern recognition to complex rule learning, with notable milestones including the development of Transformer architectures (Vaswani et al., 2017) and large-scale pretraining on diverse corpora. Early studies like Power et al. (2022) introduced the concept of grokking, where models exhibit delayed generalization, highlighting the non-linear dynamics of rule acquisition. Subsequent work by Varma et al. (2023) identified phenomena like ungrokking, where capabilities regress during training, raising questions about the stability of learned rules. Chen et al. (2024) demonstrated that training data manipulations can induce abrupt syntax transitions, emphasizing the role of data distribution. Wei et al. (2021) linked support frequency to agreement behaviors, suggesting that statistical support influences internal representations. Despite these advances, the mechanisms governing rule persistence and collapse remain poorly understood, especially the internal pathways and causal factors. This study aims to fill this gap by systematically analyzing rule dynamics, focusing on the support frequency law and displacement mechanisms, across multiple models and datasets.
Core Problem
The core challenge addressed is understanding why and how rules in neural language models can suddenly vanish or be displaced during training, despite ongoing exposure to supporting data. Traditional theories assume that rules, once learned, are stable unless explicitly forgotten, but empirical observations show that rules can abruptly collapse without data shifts or explicit forgetting signals. This phenomenon complicates efforts to ensure model reliability, interpretability, and controllability. The key questions include: • What internal mechanisms cause rules to displace or collapse? • Can we predict rule failure based on training statistics? • Is rule destruction reversible, and under what conditions? The problem is further compounded by the non-linear, emergent nature of these dynamics, which are invisible to standard loss curves, necessitating mechanistic and causal analysis to uncover the underlying principles. Addressing these issues is critical for developing models that can reliably maintain essential rules over time and across tasks.
Innovation
The primary innovations of this work include: • Introducing support frequency as a quantifiable predictor of rule survival, validated across multiple datasets and model sizes. • Demonstrating that rule collapse occurs via displacement, characterized by the zero crossing of the contrast margin (CM), rather than erasure, providing a mechanistic understanding of internal representation shifts. • Developing a causal intervention framework that manipulates support evidence in training data, establishing a monotonic dose-response relationship for rule destruction, and confirming the irreversibility of collapse. • Extending the analysis to out-of-distribution datasets, confirming the universality of displacement as a collapse mechanism. • Combining behavioral metrics, circuit-level analysis, and causal interventions to form a comprehensive mechanistic account of rule dynamics, paving the way for targeted rule management in neural models.
Methodology
- �� Construct transformer-based language models with 11.5M parameters, trained on TinyStories and a filtered web corpus, with fixed architecture, tokenizer, and training schedule. • Measure support frequency by counting occurrences of rule-supporting tokens within a 16-token window, ensuring consistency across datasets. • Evaluate model behavior using a battery of templated probes under conflict and agree conditions, with a frozen classifier to categorize outcomes into RECOVERED, DISPLACED, PARTIAL, NEVER, and UNSTABLE. • Monitor the contrast margin (CM), defined as the log-probability difference between rule-conforming and prior-conforming continuations, to track internal preference shifts. • Perform support-contradiction editing by flipping support tokens at predefined ratios (p=0.437, 0.667, 1.0), observing the dose-response effect on rule performance. • Validate the causal relationship by conducting blind, pre-registered experiments across multiple seeds, datasets, and model sizes, ensuring robustness and reproducibility. • Analyze internal circuit representations, especially last-layer heads, to identify the neural substrates of rule displacement and recovery, using ablation and reformation studies.
Experiments
- �� Multiple models ranging from 70M to 1.4B parameters are trained on TinyStories and web corpora, with systematic support frequency manipulations. • Support frequency is statistically measured and used to categorize rules as supported or unsupported, with a threshold around 0.8. • Support-contradiction editing experiments involve flipping support tokens at different ratios, observing the monotonic decline in conflict accuracy and CM. • Cross-model experiments verify that larger models exhibit shallower collapses, and the phenomena are consistent across datasets. • Out-of-distribution tests confirm that the emergence-and-collapse pattern persists beyond training data, validating the universality of displacement. • Internal circuit analysis, including head attribution and ablation, reveals that the last-layer heads encode rule preferences, and their reformation correlates with partial circuit recovery during support injections.
Results
- �� Support frequency robustly predicts rule survival: in TinyStories, rules with support >0.8 survive (9/9), while in web corpora, rules with support below this collapse (0/9). • Support-contradiction editing demonstrates a strict monotonic dose-response: increasing support flip ratio causes conflict accuracy to decline linearly from 0.94 to near zero, confirming causality. • The collapse mechanism is characterized by displacement: the contrast margin crosses zero within 100 steps, and the internal representation shifts away from the rule, favoring surface patterns. • Larger models exhibit shallower collapses, with the depth inversely related to model size, confirming the influence of capacity on rule stability. • Support injections at levels 450 times natural support fail to recover collapsed rules, emphasizing the irreversibility and non-symmetry of rule control. • Out-of-distribution experiments show the same emergence-and-collapse pattern, indicating the generality of displacement as a fundamental mechanism.
Applications
- �� The support frequency mechanism can be used to develop diagnostic tools for model safety, enabling early detection of rule collapse and targeted interventions. • In continual learning, support frequency regulation provides a pathway to maintain critical rules over long training horizons, reducing catastrophic forgetting. • For model interpretability, the contrast margin and circuit analysis offer insights into internal representations, aiding debugging and transparency. • Long-term, these findings can inform the design of more robust, controllable AI systems capable of dynamically managing their internal rules, with applications in dialogue systems, knowledge bases, and safety-critical AI deployments.
Limitations & Outlook
- �� The experiments focus on syntactic rules (e.g., pronoun gender, article allomorphy), and the applicability to more complex semantic or reasoning rules remains to be validated. • Support frequency measurement depends on specific corpus statistics and definitions, which may vary across tasks and datasets, limiting immediate generalization. • The causal interventions are limited to support editing; other mechanisms of rule collapse, such as internal representation shifts unrelated to support, are not yet explored. • The models studied are relatively small; whether the phenomena scale to larger, industrial models requires further investigation. • The irreversibility of rule collapse poses challenges for recovery strategies, emphasizing the need for proactive rule management.
Plain Language Accessible to non-experts
想象你在一家工厂工作，工厂每天都在生产各种商品。工厂里的工人们遵循一些规则，比如：如果看到红色的按钮，就按下去；如果遇到绿色的灯，就停止。随着时间推移，工厂变得越来越熟练，规则也变得越来越明确。但是，有时候，工厂会出现一种奇怪的现象：原本非常重要的规则突然被“取代”了，就像工厂的某个操作被一个新的、更快的机器取代了原有的流程。这种变化并不是因为工厂忘记了旧规则，而是被一个更受欢迎的“表面操作”所取代，导致整个生产流程发生了变化。
在深度学习模型中，类似的事情也会发生。模型在学习语言规则时，支持频率就像工厂中某个操作的频繁程度。如果某个规则经常出现，模型就会“记住”它，像工厂反复使用的操作一样；但如果这个规则变得不那么常见，模型可能会用一种“表面模式”来代替它，就像工厂用新机器取代旧流程一样。这种“取代”过程非常快，而且一旦发生，就很难再让模型重新回到原来的规则，就像工厂要恢复旧流程一样困难。这说明，模型的规则其实非常脆弱，一旦被“表面操作”取代，就很难再恢复原状。
ELI14 Explained like you're 14
想象你在学校学数学。有一些规则，比如：加法就是把两个数字放在一起，然后得到一个新数字。你学会了这个规则后，觉得它很简单，也经常用它。但是，突然有一天，老师告诉你：其实，除了加法，还有一种“表面操作”，比如用一种特殊的符号代表加法，但实际上它是另一种操作。你开始用这个新符号来做题，逐渐忘记了原来的加法规则。
在深度学习模型里，也是一样的。模型学会了某些“规则”，比如：用“she”代表女性，用“he”代表男性。但是，如果模型在训练中发现，“he”这个词更常出现，它就会用“他”这个词来代替“she”。而且，一旦模型被“表面操作”取代了原有的规则，就很难再让它回到原来的状态，就像你忘记了原来的加法一样。这说明，模型的规则其实很脆弱，一旦被“表面操作”取代，就很难再恢复原来的规则。这也是为什么我们要特别关注模型的“规则支持频率”，因为它决定了模型是否还能记住那些重要的规则。
Abstract
Midway through an ordinary pretraining run, a small language model learns the pronoun-gender rule: cued with a girl's name ("Sue cried because"), it resolves the next pronoun to she, generalizing to held-out probes (0.94 by step 925). By step 3,500 the same model scores near zero on the same probes, although the rule's evidence is still in the training data. We call this within-run reversal natural ungrokking: the corpus decides, with no trace in the loss curve, which learned rules a model keeps. Which rules survive is predictable from one corpus statistic: how often the training stream shows the rule winning. Across un-intervened runs (two corpora, three budgets, three seeds), support frequency decides a rule's fate; the data-to-parameter ratio only modulates how deeply a doomed rule falls. The same emerge-then-collapse dynamics appear in public Pythia checkpoints, collapse depth ordered by model scale as predicted. The forgetting is a displacement: a competing surface pattern out-competes the rule, and the log-probability margin between them crosses zero within 100 training steps of the behavioral collapse. Control over this fate is asymmetric: the same edit that destroys a rule on demand cannot restore it. Flipping support to counter-evidence in place kills the rule with monotone dose-response in two unrelated rules; but injecting support back, even to 450 times the level that naturally sustains it, buys no recovery. Every confirmatory threshold and prediction was pre-registered before the data it governed was read.
References (20)
Large Language Models in Machine Translation
T. Brants, Ashok Popat, P. Xu et al.
Filtered Corpus Training (FiCT) Shows that Language Models Can Generalize from Indirect Evidence
Abhinav Patil, Jaap Jumelet, Yu Ying Chiu et al.
Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models
Kushal Tirumala, Aram H. Markosyan, Luke Zettlemoyer et al.
Locating and Editing Factual Associations in GPT
Kevin Meng, David Bau, A. Andonian et al.
Scaling Data-Constrained Language Models
Niklas Muennighoff, Alexander M. Rush, B. Barak et al.
Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets
Alethea Power, Yuri Burda, Harrison Edwards et al.
An Empirical Study of Example Forgetting during Deep Neural Network Learning
Mariya Toneva, Alessandro Sordoni, Rémi Tachet des Combes et al.
OLMo: Accelerating the Science of Language Models
Dirk Groeneveld, Iz Beltagy, Pete Walsh et al.
Omnigrok: Grokking Beyond Algorithmic Data
Ziming Liu, Eric J. Michaud, Max Tegmark
How do language models learn facts? Dynamics, curricula and hallucinations
Nicolas Zucchet, Jörg Bornschein, Stephanie Chan et al.
Training Compute-Optimal Large Language Models
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch et al.
Characterizing Learning Curves During Language Model Pre-Training: Learning, Forgetting, and Stability
Tyler A. Chang, Z. Tu, B. Bergen
Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMs
Angelica Chen, Ravid Schwartz-Ziv, Kyunghyun Cho et al.
The Grammar-Learning Trajectories of Neural Language Models
Leshem Choshen, Guy Hacohen, D. Weinshall et al.
Decoupled Weight Decay Regularization
I. Loshchilov, F. Hutter
Emergent Abilities of Large Language Models
Jason Wei, Yi Tay, Rishi Bommasani et al.
Gemma 2: Improving Open Language Models at a Practical Size
Gemma Team Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa et al.
Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training
Shizhe Diao, Yu Yang, Y. Fu et al.
Progress measures for grokking via mechanistic interpretability
Neel Nanda, Lawrence Chan, Tom Lieberum et al.
Gender Bias in Neural Natural Language Processing
Kaiji Lu, Piotr (Peter) Mardziel, Fangjing Wu et al.