michalakis99/turboquant-evo2-int4

TurboQuant-Bio: int4 Evo 2 checkpoints

Pre-quantized int4 weights for Evo 2, so the 40B model runs on a single 80 GB GPU — where bf16 cannot be loaded at all.

These are quantized derivatives of Arc Institute's Evo 2 weights (arcinstitute/evo2_40b, arcinstitute/evo2_7b). All credit for the models belongs to Arc Institute; this repository contains only a compressed re-encoding of their weights.

file	size	contents
evo2_40b_int4.pt	33.8 GB	208 Linear layers int4 (65.3 → 16.8 GB, 3.88×); Hyena filters, embeddings and norms remain bf16
evo2_7b_int4.pt	5.4 GB	133 Linear layers int4

Why the file is 33.8 GB and not 16.8 GB

Only nn.Linear layers are quantized. Evo 2's Hyena filters, embeddings and norms are not Linear layers and stay bf16 — that is the remaining ~17 GB. The number that matters for fitting on a card is the 33.8 GB resident, not the Linear-only ratio.

Usage

git clone https://github.com/Georgakopoulos-Soares-lab/Turboquant_genomics.git
cd Turboquant_genomics && pip install -e .

from turboquant import load_evo2, score

model, tok = load_evo2("evo2_40b", tier="tier2",
                       int4_ckpt="evo2_40b_int4.pt", device="cuda:0")
print(score(model, tok, my_sequence, model_name="evo2_40b"))

Measured

One 80 GB GPU, 32 kb context (Evo 2's measured effective context):

config	weights resident	peak	throughput
evo2_40b bf16, 1 GPU	—	—	cannot load (OOM at block 46/50)
evo2_40b int4, 1 GPU	33.8 GB	49.0 GB	883 tok/s
evo2_40b int4, 4 GPUs	8.9 GB/GPU	18.2 GB/GPU	641 tok/s

Verified faithful: the same 32,768 bases score mean log-likelihood −0.83931 both from this checkpoint on one GPU and through the standard 4-GPU path — identical to five decimals.

One GPU is faster than four (883 vs 641 tok/s): sharding ships activations between cards at every layer, which costs more than the parallelism returns.

Important: a correctness fix comes with this

Evo 2's chunked-prefill path — the only tractable way to feed a sequence longer than one forward pass — is silently wrong upstream. It does not crash and does not warn; it returns plausible numbers essentially uncorrelated with the truth, because sequential_forward keeps only the last token of a block and that length-1 result then broadcasts over the residual stream.

The turboquant package installs the fix (install_block_continuation) automatically via load_evo2. If you drive the model yourself, you must apply it before scoring anything longer than one chunk. Details: README_chunk_prefill.md.

How much context to use

About 32 kb. Evo 2's benefit from real upstream context peaks near 32 kb and then declines — measured across 8 loci on both the 7B and 40B (p = 0.0078). Feeding 500 kb is slightly worse than feeding 32 kb, not better.

Chunk size

Use 1024 for the 40B. Chunk size — not the KV cache — drives peak memory: the Hyena modal-FFT buffer is ≈8.6 GB at chunk 4096, so 4096 and 8192 both OOM on an 80 GB card while 1024 peaks at 49 GB. Fidelity is independent of chunk size.