Hugging Face's logo
Syed Affan's picture

Syed Affan

sulpha
3 5 9
reaperdoesntknow's profile picture hmb's profile picture
Β·

AI & ML interests

None yet

Recent Activity

liked a model about 1 month ago
yuxinlu1/gemma-4-12B-coder-fable5-composer2.5-v1-GGUF
reacted to reaperdoesntknow's post with πŸ‘€ 4 months ago
# Three Teachers, One Student: Dual-Cognition Reasoning at 1.7B We distilled Qwen3-30B-A3B into 1.7B students that critique their own reasoning. H100, BF16, Apache 2.0. Here's our pipeline. **Stage 1 — Three Teachers, Three Profiles.** Same 30B base, three variants: Instruct (structured output), Thinking (extended deliberation), Coder (STEM decomposition). Each distillation uses proof-weighted KD — 2.25× amplified loss on reasoning tokens, decaying to 1.1×. The student learns *where to think harder*, not just what to output. **Stage 2 — Topology-Aware KD (TKD).** Standard KD treats the teacher's distribution as smooth. Language isn't smooth — it has topic shifts, reasoning pivots, register changes. We use Discrepancy Calculus to detect these structural boundaries, then amplify loss at jumps (3σ threshold) and cut training windows at low-discrepancy positions. The student preserves the teacher's structural knowledge, not just surface statistics. **Stage 3 — Ghost Imprinting.** Sequential distillation from different teachers leaves residual fields in weight space that neither teacher put there individually. The Cantor component of BV decomposition, applied to parameters. Models distilled Thinking→Coder exhibit deliberation patterns from the Thinking teacher that survived Coder overwriting. Emergent capability from structural residuals. **Stage 4 — DualMind.** One model, two voices, shared weights: ``` <explore> — free derivation, speculation <examine> — adversarial self-critique <response> — clean synthesis ``` The multi-model collision array collapsed into a single architecture. Role tokens, no extra parameters. For the full method: https://huggingface.co/reaperdoesntknow/DualMind_Methodolgy doi:10.57967/hf/8184.
upvoted an article 10 months ago
Smol2Operator: Post-Training GUI Agents for Computer Use
View all activity

Organizations

scikit-learn's profile picture VidhyaDojo's profile picture