| --- |
| license: apache-2.0 |
| tags: |
| - pytorch |
| - transformer |
| - mamba |
| - moe |
| - hybrid |
| - matryoshka |
| - gpt-oss |
| - adaptive-compute |
| pipeline_tag: text-generation |
| --- |
| |
| # 🌀 GPT-OSS Adamba: Hybrid MoE + Mamba |
|
|
| > **21.9B** parameters | **32 experts** | **Mamba-enhanced** reasoning backbone |
|
|
| 📂 **[GitHub](https://github.com/unixsysdev/adamba)** | 🤗 **[Original Adamba](https://huggingface.co/datasysdev/adamba)** |
|
|
| ## Available Checkpoints |
|
|
| | Variant | Parameters | Dim | Features | Status | Download | |
| |---------|------------|-----|----------|--------|----------| |
| | gptoss_phase1 | 21.9B | 2880 | mamba_integration, moe_32experts | ✅ | [Download](./checkpoints/gptoss_phase1.pt) | |
| | gptoss_phase2 | 21.9B | 2880 | matryoshka, early_exit, moe_32experts | ⏳ | — | |
| | gptoss_phase3 | 30B+ | 4096 | matryoshka, early_exit, moe_32experts, expansion | ⏳ | — | |
| | gptoss_sft | 21.9B | 2880 | matryoshka, moe_32experts, sft | ⏳ | — | |
| |
| ## Architecture |
| |
| Built on [OpenAI GPT-OSS 20B](https://huggingface.co/openai/gpt-oss-20b) with Mamba integration: |
| |
| | Component | Spec | |
| |-----------|------| |
| | **Base Model** | GPT-OSS 20B MoE | |
| | **Hidden Dim** | 2880 | |
| | **Attention** | 24 layers (sliding + full alternating) | |
| | **Mamba** | 12 layers (interleaved 2:1) | |
| | **MoE** | 32 experts, top-4 routing | |
| | **Vocab** | 201,088 tokens | |
| | **Total Blocks** | 36 (24 Attn + 12 Mamba) | |
| |
| ``` |
| ┌─────────────────────────────────────────────────────┐ |
| │ GPT-OSS 20B (Attention + MoE) │ |
| │ ↓ Surgery (inject 12 Mamba layers) │ |
| │ Hybrid: A-A-M-A-A-M-... pattern │ |
| │ ↓ Phase 1 (train Mamba only) │ |
| │ Mamba learns to "speak GPT-OSS language" │ |
| │ ↓ Phase 2 (enable Matryoshka) │ |
| │ Adaptive compute: 128 → 2880 dim per layer │ |
| └─────────────────────────────────────────────────────┘ |
| ``` |
| |
| ## Training Status |
| |
| **Phase 1**: Mamba integration (freeze Attention+MoE, train Mamba) |
| |
| ## Usage |
| |
| ```python |
| # Coming soon - inference code |
| # See: https://github.com/unixsysdev/adamba |
| ``` |
| |
| ## License |
| |
| Apache 2.0 (same as GPT-OSS) |
| |
