mlx-community/Fara-7B-4bit

microsoft/Fara-7B converted to MLX and quantized to 4-bit, for inference on Apple Silicon.

Fara-7B is a computer-use agent model built on Qwen2.5-VL — it reads screenshots and acts on interfaces. The vision path is preserved in this conversion, which for this model class is the point.

See also Fara-7B-8bit for the higher-fidelity variant, and Fara1.5-9B-8bit for the newer generation of the same family.

Quantization

Requested bits	4
Group size	64
Mode	affine
Effective bits per weight	5.44
On-disk size	5.3 GB

Effective bits exceed the requested value because mlx-vlm quantizes only the language model and leaves the vision tower in bf16 by design — 390 vision tensors, none of them quantized. The vision encoder is a small share of the weights but disproportionately sensitive to quantization error.

language_model : 198 tensors quantized (4-bit, group size 64)
vision_tower   : 390 tensors, bf16    <- unquantized

Fidelity vs the original weights

Measured against the bf16 source, tensor by tensor, over all 198 quantized tensors (7,615,283,200 parameters). No prompts or sampling involved — this is a direct measurement of how much numerical information the quantization discarded, and it is exact and hardware-independent.

Metric	4-bit	8-bit
Relative L2 error	9.38%	0.74%
Cosine similarity	0.995603	0.999973
Signal-to-quantization-noise	20.55 dB	42.66 dB
Worst single-element error	0.089844	0.007812

Highest-error tensors at 4-bit — v_proj and early-layer down_proj are consistently the most sensitive:

rel_l2=0.10771  snr= 19.36 dB  language_model.model.layers.1.mlp.down_proj
rel_l2=0.10725  snr= 19.39 dB  language_model.model.layers.23.self_attn.v_proj
rel_l2=0.10547  snr= 19.54 dB  language_model.model.layers.22.self_attn.v_proj
rel_l2=0.10419  snr= 19.64 dB  language_model.model.layers.25.self_attn.v_proj
rel_l2=0.10355  snr= 19.70 dB  language_model.model.layers.27.self_attn.v_proj

Going from 4-bit to 8-bit improves the signal-to-quantization-noise ratio by 22 dB — a factor of roughly 160 in noise power. If your machine has the memory, prefer the 8-bit variant; use this one when 5.3 GB versus 8.8 GB is the deciding constraint.

Throughput

Measured on an M2 Pro / 32 GB, 64 generated tokens, greedy.

Variant	Decode tok/s	Prompt tok/s	Peak RAM
4-bit	36.0	111.6	5.80 GB
8-bit	18.8	105.5	9.57 GB

This variant decodes 1.9x faster at 1.7x less memory. Numbers do not transfer across chips.

Why there is no behavioural evaluation

Other conversions in this series report perplexity ratio, top-1 agreement and KL divergence against the bf16 source — see Fara1.5-9B-8bit, which reaches top-1 agreement of 1.000 that way. That protocol does not work for Fara-7B, and the reason is worth stating rather than quietly omitting.

Fara-7B is a computer-use model: it expects a screenshot plus an action space, not prose. Scored on plain text it is out of distribution before any quantization — the unquantized bf16 source itself has a perplexity of 12.30 on the same passages where Fara1.5-9B scores 3.23. With a distribution that flat, the metric stops discriminating: measured that way this 4-bit variant scored a better perplexity ratio (1.2312) and KL (0.2768) than the 8-bit one (1.4897 / 0.4809), which is impossible — a 4-bit quantization cannot be more faithful than an 8-bit one of the same model.

The weight-level numbers above give the correct ordering, so the anomaly is in the measurement, not in the weights. Those behavioural figures are therefore excluded rather than reported.

A meaningful behavioural benchmark for this model would need screenshots and a verifiable action space — a computer-use harness, which was not available here.

What was not measured

No standard benchmarks: no ScreenSpot, WebArena, OSWorld, or any agentic evaluation. No judged quality. The vision path was verified to load and run, not scored on a dataset. If your use case is the full computer-use loop, evaluate on your own tasks.

Usage

pip install mlx-vlm

from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template

model, processor = load("mlx-community/Fara-7B-4bit")

prompt = apply_chat_template(
    processor, model.config,
    "Describe this screenshot. What buttons do you see?",
    num_images=1,
)
out = generate(model, processor, prompt, image=["screenshot.png"], max_tokens=256)
print(out.text)

Text-only works too — pass num_images=0 and omit image.

Note that stock mlx-lm loads the text path only; use mlx-vlm for image input.

Credits

All credit for the model belongs to Microsoft. This is a format conversion and quantization; no training or fine-tuning was performed. Licensed MIT, as the original. See the original card for intended use and limitations.