Can Small Language Models Learn From Bigger Ones Without Copying Their Mistakes?

Massive AI foundation models like GPT-4, Claude 3.5, and Llama 3 70B possess impressive reasoning capabilities, but they require massive cloud server farms, hundreds of expensive GPUs, and immense electrical power to run. To bring AI directly onto smartphones, laptops, and edge devices, researchers rely on a core model compression technique called Knowledge Distillation—training small "student" models (typically 1B to 7B parameters) using the outputs of giant "teacher" models. But can small models acquire true reasoning from big ones without inheriting—and magnifying—their mistakes?

Investigating Model Compression and Synthetic Data

Computer science researchers at UC Berkeley, Tsinghua University, MIT, and Meta AI set out to investigate the deep trade-offs of knowledge distillation. When a small student model is trained on millions of synthetic sentences generated by a large teacher model, does the student actually acquire true underlying logic, or does it merely learn to imitate the teacher's superficial writing style? More importantly, researchers analyzed what happens to error rates and hallucination propagation when small models learn from synthetic data generated by flawed teachers.

Experimental Setup: Testing Student Models

To evaluate distilled models, research teams led by Arnab Gudibande (UC Berkeley) and Yuxian Gu (Tsinghua / Microsoft Research) trained small student models across two primary distillation methodologies:

• Off-Policy Imitation Training: Feeding student models static datasets composed of millions of synthetic Q&A responses generated by teacher models like GPT-4.

• On-Policy Distillation (MiniLLM / Reverse-KL): Allowing the small student model to generate its own trial answers dynamically, while receiving continuous mathematical feedback from the teacher model's probability distribution.

The researchers benchmarked these compact student models against their foundation teacher models across factual recall, mathematical reasoning (GSM8K), coding logic (HumanEval), and overall hallucination rates.

What Researchers Found: The "Imitation Trap"

The experimental findings uncovered a critical vulnerability in standard distillation known as The Imitation Trap:

When small student models are trained purely on synthetic text generated by large teacher models, they become highly adept at copying the teacher's tone, formatting, and stylistic confidence. They sound just as smart, articulate, and authoritative as GPT-4. However, when tested on hard factual questions or novel reasoning problems outside the training set, their factual accuracy plummets.

Because small models have far fewer parameters, they lack the raw memory capacity to store the vast world facts that the teacher model relies on. As a result, imitation models become overconfident hallucination repeaters—generating false claims with the polished, authoritative style of their teacher.

The On-Policy Solution: Cutting Hallucinations by 35%

To break out of the imitation trap, researchers developed On-Policy Distillation (MiniLLM). Instead of forcing the student to passively memorize the teacher's synthetic text outputs, on-policy distillation forces the student model to generate its own answers first. The teacher model then evaluates the student's output and penalizes the student only where its probability distribution diverges.

By training student models on their own generations rather than passive teacher outputs, researchers achieved major breakthroughs:

• Hallucination Drop: Reduced small model hallucination rates by 35%.

• Reasoning Boost: Increased logical reasoning performance by 22% on standard benchmarks like GSM8K and HumanEval.

Limitations: Tail Knowledge Loss and Synthetic Pollution

Despite these algorithmic improvements, fundamental structural limits remain in on-device AI:

• Tail Knowledge Loss: A 3B parameter model running on a mobile phone simply cannot store the long-tail factual knowledge (such as obscure historical dates or specialized medical facts) contained in a 1.75-trillion parameter foundation model.

• Synthetic Data Pollution: If a teacher model contains subtle biases or hallucinations during synthetic data generation, the student model absorbs those errors as absolute ground truth, amplifying mistakes across downstream deployments.

Industrial Impact and Next Steps for On-Device AI

Knowledge distillation is essential for privacy-focused, low-latency, on-device AI. As smartphone manufacturers, laptop makers, and automotive companies deploy local AI models that operate entirely without internet connections, understanding distillation limits ensures that small models remain accurate, efficient, and reliable without overpromising their capabilities.