mlx-community/Fara-7B-8bit

microsoft/Fara-7B converted to MLX and quantized to 8-bit, for inference on Apple Silicon.

Fara-7B is a computer-use agent model built on Qwen2.5-VL — it reads screenshots and acts on interfaces. The vision path is preserved in this conversion, which for this model class is the point.

See also Fara-7B-4bit for the smaller variant, and Fara1.5-9B-8bit for the newer generation of the same family.

Quantization

Requested bits	8
Group size	64
Mode	affine
Effective bits per weight	9.11
On-disk size	8.8 GB
Shards	2

Effective bits exceed the requested value because mlx-vlm quantizes only the language model and leaves the vision tower in bf16 by design — 390 vision tensors, none of them quantized. The vision encoder is a small share of the weights but disproportionately sensitive to quantization error.

language_model : 198 tensors quantized (8-bit, group size 64)
vision_tower   : 390 tensors, bf16    <- unquantized

Fidelity vs the original weights

Measured against the bf16 source, tensor by tensor, over all 198 quantized tensors (7,615,283,200 parameters). No prompts or sampling involved — this is a direct measurement of how much numerical information the quantization discarded, and it is exact and hardware-independent.

Metric	8-bit	4-bit
Relative L2 error	0.74%	9.38%
Cosine similarity	0.999973	0.995603
Signal-to-quantization-noise	42.66 dB	20.55 dB
Worst single-element error	0.007812	0.089844

Highest-error tensors at 8-bit — v_proj and early-layer down_proj are consistently the most sensitive:

rel_l2=0.00870  snr= 41.21 dB  language_model.model.layers.1.mlp.down_proj
rel_l2=0.00868  snr= 41.23 dB  language_model.model.layers.23.self_attn.v_proj
rel_l2=0.00848  snr= 41.44 dB  language_model.model.layers.22.self_attn.v_proj
rel_l2=0.00847  snr= 41.44 dB  language_model.model.layers.25.self_attn.v_proj
rel_l2=0.00824  snr= 41.68 dB  language_model.lm_head

Throughput

Measured on an M2 Pro / 32 GB, 64 generated tokens, greedy.

Variant	Decode tok/s	Prompt tok/s	Peak RAM
8-bit	18.8	105.5	9.57 GB
4-bit	36.0	111.6	5.80 GB

The 4-bit variant decodes 1.9x faster at 1.7x less memory, at the cost of the fidelity difference shown above (20.55 dB vs 42.66 dB). Numbers do not transfer across chips.

Why there is no behavioural evaluation

Other conversions in this series report perplexity ratio, top-1 agreement and KL divergence against the bf16 source — see Fara1.5-9B-8bit, which reaches top-1 agreement of 1.000 that way. That protocol does not work for Fara-7B, and the reason is worth stating rather than quietly omitting.

Fara-7B is a computer-use model: it expects a screenshot plus an action space, not prose. Scored on plain text it is out of distribution before any quantization — the unquantized bf16 source itself has a perplexity of 12.30 on the same passages where Fara1.5-9B scores 3.23. With a distribution that flat, the metric stops discriminating. Measured that way, the 4-bit variant came out better than the 8-bit one:

Text-only teacher forcing	8-bit	4-bit
Perplexity ratio	1.4897	1.2312
KL (nats/token)	0.4809	0.2768

That ordering is impossible — a 4-bit quantization cannot be more faithful than an 8-bit one of the same model. The weight-level numbers above confirm the correct ordering (42.66 dB vs 20.55 dB), so the anomaly is in the measurement, not in the weights. Reporting those behavioural figures would have been misleading, so they are excluded and the exact weight-level comparison is used instead.

A meaningful behavioural benchmark for this model would need screenshots and a verifiable action space — a computer-use harness, which was not available here.

What was not measured

No standard benchmarks: no ScreenSpot, WebArena, OSWorld, or any agentic evaluation. No judged quality. The vision path was verified to load and run, not scored on a dataset. If your use case is the full computer-use loop, evaluate on your own tasks.

Usage

pip install mlx-vlm

from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template

model, processor = load("mlx-community/Fara-7B-8bit")

prompt = apply_chat_template(
    processor, model.config,
    "Describe this screenshot. What buttons do you see?",
    num_images=1,
)
out = generate(model, processor, prompt, image=["screenshot.png"], max_tokens=256)
print(out.text)

Text-only works too — pass num_images=0 and omit image.

Note that stock mlx-lm loads the text path only; use mlx-vlm for image input.

Credits

All credit for the model belongs to Microsoft. This is a format conversion and quantization; no training or fine-tuning was performed. Licensed MIT, as the original. See the original card for intended use and limitations.