The Geometric Alignment Tax: Tokenization vs. Continuous Geometry in Scientific Foundation Models

TL;DR: Biological and physical foundation models systematically fail to preserve the continuous geometry of the natural world. The root cause? Forcing continuous physical manifolds through discrete categorical bottlenecks (tokenization + cross-entropy loss).
Key Highlights Across 14 Models:
• The Causal Proof: Swapping a discrete cross-entropy head for a continuous MSE head on identical encoders reduces geometric distortion by up to 8.5x.
• Scaling Makes It Worse: Evaluating the ESM-2 suite shows geometric stability declines from 8M to 3B parameters. The apparent "recovery" at 15B is actually coherent global manifold drift.
• The Evo 2 Illusion: A controlled ablation proves Evo 2's (7B) reverse-complement robustness on real DNA is an artifact of sequence-level k-mer histogram matching, not learned biophysics.
• The Information-Theoretic Trap: Using MINE and rate-distortion theory, we prove models inevitably fall into one of three failure regimes: Local-Global Decoupling (Evo 2), Representational Compression (OpenFold), or Geometric Vacuity (ProtMamba).
The Takeaway: Scale is not all you need for AI for Science. To build true Scientific AGI, we must move beyond the discrete tokenization playbook and design architectures that natively preserve continuous physical invariants.